2023-04-06 18:56:09,246 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67 2023-04-06 18:56:09,258 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-06 18:56:09,271 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-06 18:56:09,271 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc, deleteOnExit=true 2023-04-06 18:56:09,271 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-06 18:56:09,272 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/test.cache.data in system properties and HBase conf 2023-04-06 18:56:09,272 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/hadoop.tmp.dir in system properties and HBase conf 2023-04-06 18:56:09,273 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/hadoop.log.dir in system properties and HBase conf 2023-04-06 18:56:09,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-06 18:56:09,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-06 18:56:09,274 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-06 18:56:09,365 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-06 18:56:09,670 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-06 18:56:09,674 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-06 18:56:09,674 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-06 18:56:09,674 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-06 18:56:09,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 18:56:09,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-06 18:56:09,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-06 18:56:09,675 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 18:56:09,676 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 18:56:09,676 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-06 18:56:09,676 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/nfs.dump.dir in system properties and HBase conf 2023-04-06 18:56:09,676 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/java.io.tmpdir in system properties and HBase conf 2023-04-06 18:56:09,677 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 18:56:09,677 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-06 18:56:09,677 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-06 18:56:10,069 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 18:56:10,072 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 18:56:10,285 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-06 18:56:10,431 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-06 18:56:10,444 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 18:56:10,474 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 18:56:10,500 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/java.io.tmpdir/Jetty_localhost_localdomain_37113_hdfs____ec6wp5/webapp 2023-04-06 18:56:10,606 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:37113 2023-04-06 18:56:10,638 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 18:56:10,639 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 18:56:10,947 WARN [Listener at localhost.localdomain/34815] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 18:56:11,033 WARN [Listener at localhost.localdomain/34815] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-06 18:56:11,049 WARN [Listener at localhost.localdomain/34815] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 18:56:11,055 INFO [Listener at localhost.localdomain/34815] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 18:56:11,060 INFO [Listener at localhost.localdomain/34815] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/java.io.tmpdir/Jetty_localhost_37735_datanode____.e718kk/webapp 2023-04-06 18:56:11,137 INFO [Listener at localhost.localdomain/34815] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37735 2023-04-06 18:56:11,419 WARN [Listener at localhost.localdomain/41219] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 18:56:11,773 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf563b6610f7428fa: Processing first storage report for DS-27d14085-c0ab-41d2-965d-ad0f154369c2 from datanode c96f5453-3775-4f27-baac-cfdbfa27cff3 2023-04-06 18:56:11,774 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf563b6610f7428fa: from storage DS-27d14085-c0ab-41d2-965d-ad0f154369c2 node DatanodeRegistration(127.0.0.1:40017, datanodeUuid=c96f5453-3775-4f27-baac-cfdbfa27cff3, infoPort=36679, infoSecurePort=0, ipcPort=41219, storageInfo=lv=-57;cid=testClusterID;nsid=1642455155;c=1680807370127), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-06 18:56:11,775 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf563b6610f7428fa: Processing first storage report for DS-b0cd12ce-52dc-4a46-90d0-c090ff66b686 from datanode c96f5453-3775-4f27-baac-cfdbfa27cff3 2023-04-06 18:56:11,775 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf563b6610f7428fa: from storage DS-b0cd12ce-52dc-4a46-90d0-c090ff66b686 node DatanodeRegistration(127.0.0.1:40017, datanodeUuid=c96f5453-3775-4f27-baac-cfdbfa27cff3, infoPort=36679, infoSecurePort=0, ipcPort=41219, storageInfo=lv=-57;cid=testClusterID;nsid=1642455155;c=1680807370127), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-06 18:56:11,930 DEBUG [Listener at localhost.localdomain/41219] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67 2023-04-06 18:56:11,987 INFO [Listener at localhost.localdomain/41219] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc/zookeeper_0, clientPort=63695, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-06 18:56:11,998 INFO [Listener at localhost.localdomain/41219] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63695 2023-04-06 18:56:12,005 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:12,007 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:12,612 INFO [Listener at localhost.localdomain/41219] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356 with version=8 2023-04-06 18:56:12,612 INFO [Listener at localhost.localdomain/41219] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/hbase-staging 2023-04-06 18:56:12,896 INFO [Listener at localhost.localdomain/41219] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-06 18:56:13,318 INFO [Listener at localhost.localdomain/41219] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 18:56:13,348 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,349 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,349 INFO [Listener at localhost.localdomain/41219] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 18:56:13,349 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,349 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 18:56:13,498 INFO [Listener at localhost.localdomain/41219] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 18:56:13,567 DEBUG [Listener at localhost.localdomain/41219] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-06 18:56:13,656 INFO [Listener at localhost.localdomain/41219] ipc.NettyRpcServer(120): Bind to /148.251.75.209:45203 2023-04-06 18:56:13,668 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:13,670 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:13,690 INFO [Listener at localhost.localdomain/41219] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45203 connecting to ZooKeeper ensemble=127.0.0.1:63695 2023-04-06 18:56:13,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:452030x0, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 18:56:13,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45203-0x10087a33cf40000 connected 2023-04-06 18:56:13,763 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 18:56:13,764 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 18:56:13,767 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 18:56:13,773 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45203 2023-04-06 18:56:13,773 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45203 2023-04-06 18:56:13,774 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45203 2023-04-06 18:56:13,776 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45203 2023-04-06 18:56:13,778 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45203 2023-04-06 18:56:13,783 INFO [Listener at localhost.localdomain/41219] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356, hbase.cluster.distributed=false 2023-04-06 18:56:13,849 INFO [Listener at localhost.localdomain/41219] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 18:56:13,849 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,849 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,850 INFO [Listener at localhost.localdomain/41219] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 18:56:13,850 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 18:56:13,850 INFO [Listener at localhost.localdomain/41219] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 18:56:13,854 INFO [Listener at localhost.localdomain/41219] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 18:56:13,857 INFO [Listener at localhost.localdomain/41219] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34607 2023-04-06 18:56:13,859 INFO [Listener at localhost.localdomain/41219] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-06 18:56:13,864 DEBUG [Listener at localhost.localdomain/41219] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-06 18:56:13,865 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:13,868 INFO [Listener at localhost.localdomain/41219] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:13,870 INFO [Listener at localhost.localdomain/41219] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34607 connecting to ZooKeeper ensemble=127.0.0.1:63695 2023-04-06 18:56:13,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:346070x0, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 18:56:13,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34607-0x10087a33cf40001 connected 2023-04-06 18:56:13,875 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 18:56:13,876 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 18:56:13,877 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 18:56:13,877 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34607 2023-04-06 18:56:13,878 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34607 2023-04-06 18:56:13,878 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34607 2023-04-06 18:56:13,878 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34607 2023-04-06 18:56:13,879 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34607 2023-04-06 18:56:13,880 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:13,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 18:56:13,889 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:13,907 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 18:56:13,907 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 18:56:13,907 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:13,908 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 18:56:13,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 18:56:13,910 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,45203,1680807372735 from backup master directory 2023-04-06 18:56:13,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:13,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 18:56:13,920 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 18:56:13,920 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:13,922 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-06 18:56:13,923 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-06 18:56:14,016 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/hbase.id with ID: 8d1549ca-33b1-4f71-a87b-08c28ecd295f 2023-04-06 18:56:14,062 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 18:56:14,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:14,115 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x38302e48 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:14,152 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c892a08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:14,176 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:14,178 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-06 18:56:14,195 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-06 18:56:14,195 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-06 18:56:14,197 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:14,201 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:14,202 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 18:56:14,238 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store-tmp 2023-04-06 18:56:14,667 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:14,667 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 18:56:14,667 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:56:14,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:56:14,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 18:56:14,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:56:14,668 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:56:14,668 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 18:56:14,670 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/WALs/jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:14,699 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C45203%2C1680807372735, suffix=, logDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/WALs/jenkins-hbase20.apache.org,45203,1680807372735, archiveDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/oldWALs, maxLogs=10 2023-04-06 18:56:14,764 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK] 2023-04-06 18:56:14,771 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:14,833 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/WALs/jenkins-hbase20.apache.org,45203,1680807372735/jenkins-hbase20.apache.org%2C45203%2C1680807372735.1680807374711 2023-04-06 18:56:14,834 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK]] 2023-04-06 18:56:14,835 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:14,835 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:14,838 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,840 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,891 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,899 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-06 18:56:14,928 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-06 18:56:14,941 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:14,947 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,949 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,965 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 18:56:14,970 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:56:14,971 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64400580, jitterRate=-0.040356576442718506}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 18:56:14,971 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 18:56:14,973 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-06 18:56:14,996 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-06 18:56:14,997 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-06 18:56:15,000 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-06 18:56:15,002 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-06 18:56:15,033 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-06 18:56:15,034 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-06 18:56:15,057 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-06 18:56:15,061 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-06 18:56:15,087 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-06 18:56:15,092 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-06 18:56:15,094 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-06 18:56:15,098 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-06 18:56:15,102 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-06 18:56:15,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:15,105 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-06 18:56:15,105 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-06 18:56:15,117 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-06 18:56:15,121 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 18:56:15,121 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 18:56:15,121 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:15,121 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,45203,1680807372735, sessionid=0x10087a33cf40000, setting cluster-up flag (Was=false) 2023-04-06 18:56:15,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:15,141 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-06 18:56:15,142 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:15,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:15,156 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-06 18:56:15,157 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:15,160 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.hbase-snapshot/.tmp 2023-04-06 18:56:15,182 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(952): ClusterId : 8d1549ca-33b1-4f71-a87b-08c28ecd295f 2023-04-06 18:56:15,187 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-06 18:56:15,192 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-06 18:56:15,192 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-06 18:56:15,195 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-06 18:56:15,196 DEBUG [RS:0;jenkins-hbase20:34607] zookeeper.ReadOnlyZKClient(139): Connect 0x261697dd to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:15,203 DEBUG [RS:0;jenkins-hbase20:34607] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28d17d6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:15,204 DEBUG [RS:0;jenkins-hbase20:34607] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ceefcad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 18:56:15,234 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34607 2023-04-06 18:56:15,238 INFO [RS:0;jenkins-hbase20:34607] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-06 18:56:15,238 INFO [RS:0;jenkins-hbase20:34607] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-06 18:56:15,238 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1023): About to register with Master. 2023-04-06 18:56:15,240 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,45203,1680807372735 with isa=jenkins-hbase20.apache.org/148.251.75.209:34607, startcode=1680807373848 2023-04-06 18:56:15,254 DEBUG [RS:0;jenkins-hbase20:34607] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-06 18:56:15,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-06 18:56:15,275 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 18:56:15,275 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 18:56:15,275 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 18:56:15,276 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 18:56:15,276 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-06 18:56:15,276 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,276 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 18:56:15,276 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,286 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680807405286 2023-04-06 18:56:15,289 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-06 18:56:15,291 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 18:56:15,292 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-06 18:56:15,298 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:15,301 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-06 18:56:15,309 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52451, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-06 18:56:15,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-06 18:56:15,310 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-06 18:56:15,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-06 18:56:15,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-06 18:56:15,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,314 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-06 18:56:15,316 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-06 18:56:15,316 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-06 18:56:15,319 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-06 18:56:15,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:15,320 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-06 18:56:15,321 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680807375321,5,FailOnTimeoutGroup] 2023-04-06 18:56:15,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680807375322,5,FailOnTimeoutGroup] 2023-04-06 18:56:15,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,322 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-06 18:56:15,323 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,324 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,338 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:15,339 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:15,339 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356 2023-04-06 18:56:15,350 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(2829): Master is not running yet 2023-04-06 18:56:15,351 WARN [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-06 18:56:15,355 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:15,358 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 18:56:15,360 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/info 2023-04-06 18:56:15,361 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 18:56:15,362 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,363 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 18:56:15,365 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/rep_barrier 2023-04-06 18:56:15,366 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 18:56:15,367 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,367 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 18:56:15,369 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/table 2023-04-06 18:56:15,369 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 18:56:15,370 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,371 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740 2023-04-06 18:56:15,372 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740 2023-04-06 18:56:15,376 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:56:15,378 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 18:56:15,382 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:56:15,383 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60545966, jitterRate=-0.0977948009967804}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:56:15,383 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 18:56:15,383 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 18:56:15,383 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 18:56:15,383 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 18:56:15,383 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 18:56:15,383 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 18:56:15,384 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 18:56:15,384 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 18:56:15,389 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 18:56:15,389 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-06 18:56:15,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-06 18:56:15,408 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-06 18:56:15,410 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-06 18:56:15,452 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,45203,1680807372735 with isa=jenkins-hbase20.apache.org/148.251.75.209:34607, startcode=1680807373848 2023-04-06 18:56:15,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,465 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356 2023-04-06 18:56:15,465 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:34815 2023-04-06 18:56:15,465 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-06 18:56:15,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 18:56:15,470 DEBUG [RS:0;jenkins-hbase20:34607] zookeeper.ZKUtil(162): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,470 WARN [RS:0;jenkins-hbase20:34607] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 18:56:15,470 INFO [RS:0;jenkins-hbase20:34607] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 18:56:15,471 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/WALs/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,471 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34607,1680807373848] 2023-04-06 18:56:15,479 DEBUG [RS:0;jenkins-hbase20:34607] zookeeper.ZKUtil(162): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,491 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-06 18:56:15,501 INFO [RS:0;jenkins-hbase20:34607] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-06 18:56:15,522 INFO [RS:0;jenkins-hbase20:34607] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-06 18:56:15,528 INFO [RS:0;jenkins-hbase20:34607] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-06 18:56:15,528 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,529 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-06 18:56:15,537 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,537 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,538 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,538 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,538 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,538 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,539 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 18:56:15,539 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,539 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,539 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,539 DEBUG [RS:0;jenkins-hbase20:34607] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 18:56:15,540 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,540 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,540 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,555 INFO [RS:0;jenkins-hbase20:34607] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-06 18:56:15,557 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34607,1680807373848-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:15,562 DEBUG [jenkins-hbase20:45203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-06 18:56:15,565 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34607,1680807373848, state=OPENING 2023-04-06 18:56:15,570 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-06 18:56:15,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:15,572 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 18:56:15,574 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:15,577 INFO [RS:0;jenkins-hbase20:34607] regionserver.Replication(203): jenkins-hbase20.apache.org,34607,1680807373848 started 2023-04-06 18:56:15,577 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,34607,1680807373848, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34607, sessionid=0x10087a33cf40001 2023-04-06 18:56:15,577 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-06 18:56:15,577 DEBUG [RS:0;jenkins-hbase20:34607] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,578 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34607,1680807373848' 2023-04-06 18:56:15,578 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:15,578 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34607,1680807373848' 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-06 18:56:15,579 DEBUG [RS:0;jenkins-hbase20:34607] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-06 18:56:15,580 DEBUG [RS:0;jenkins-hbase20:34607] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-06 18:56:15,580 INFO [RS:0;jenkins-hbase20:34607] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-06 18:56:15,580 INFO [RS:0;jenkins-hbase20:34607] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-06 18:56:15,690 INFO [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34607%2C1680807373848, suffix=, logDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/WALs/jenkins-hbase20.apache.org,34607,1680807373848, archiveDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/oldWALs, maxLogs=32 2023-04-06 18:56:15,708 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK] 2023-04-06 18:56:15,711 INFO [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/WALs/jenkins-hbase20.apache.org,34607,1680807373848/jenkins-hbase20.apache.org%2C34607%2C1680807373848.1680807375692 2023-04-06 18:56:15,712 DEBUG [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK]] 2023-04-06 18:56:15,757 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:15,760 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-06 18:56:15,764 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-06 18:56:15,776 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-06 18:56:15,777 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 18:56:15,780 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34607%2C1680807373848.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/WALs/jenkins-hbase20.apache.org,34607,1680807373848, archiveDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/oldWALs, maxLogs=32 2023-04-06 18:56:15,795 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK] 2023-04-06 18:56:15,799 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/WALs/jenkins-hbase20.apache.org,34607,1680807373848/jenkins-hbase20.apache.org%2C34607%2C1680807373848.meta.1680807375782.meta 2023-04-06 18:56:15,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40017,DS-27d14085-c0ab-41d2-965d-ad0f154369c2,DISK]] 2023-04-06 18:56:15,802 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:15,804 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-06 18:56:15,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-06 18:56:15,827 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-06 18:56:15,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-06 18:56:15,833 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:15,833 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-06 18:56:15,833 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-06 18:56:15,836 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 18:56:15,838 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/info 2023-04-06 18:56:15,838 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/info 2023-04-06 18:56:15,839 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 18:56:15,840 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,840 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 18:56:15,841 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/rep_barrier 2023-04-06 18:56:15,841 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/rep_barrier 2023-04-06 18:56:15,842 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 18:56:15,843 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,843 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 18:56:15,844 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/table 2023-04-06 18:56:15,845 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/table 2023-04-06 18:56:15,845 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 18:56:15,846 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:15,848 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740 2023-04-06 18:56:15,850 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740 2023-04-06 18:56:15,854 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:56:15,856 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 18:56:15,858 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60979875, jitterRate=-0.09132905304431915}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:56:15,858 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 18:56:15,868 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680807375750 2023-04-06 18:56:15,886 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-06 18:56:15,887 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-06 18:56:15,888 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34607,1680807373848, state=OPEN 2023-04-06 18:56:15,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-06 18:56:15,890 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 18:56:15,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-06 18:56:15,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34607,1680807373848 in 316 msec 2023-04-06 18:56:15,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-06 18:56:15,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 498 msec 2023-04-06 18:56:15,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 696 msec 2023-04-06 18:56:15,905 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680807375905, completionTime=-1 2023-04-06 18:56:15,906 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-06 18:56:15,906 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-06 18:56:15,969 DEBUG [hconnection-0x6bd95a05-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:15,973 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:15,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-06 18:56:15,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680807435988 2023-04-06 18:56:15,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680807495988 2023-04-06 18:56:15,988 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 82 msec 2023-04-06 18:56:16,012 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45203,1680807372735-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:16,013 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45203,1680807372735-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:16,013 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45203,1680807372735-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:16,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:45203, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:16,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-06 18:56:16,020 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-06 18:56:16,030 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-06 18:56:16,031 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:16,041 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-06 18:56:16,044 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:56:16,046 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:56:16,067 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:16,070 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af empty. 2023-04-06 18:56:16,070 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:16,071 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-06 18:56:16,495 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:16,497 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c1504e0a89a4202f82012694b63b21af, NAME => 'hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:56:16,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:16,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c1504e0a89a4202f82012694b63b21af, disabling compactions & flushes 2023-04-06 18:56:16,909 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:16,909 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:16,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. after waiting 0 ms 2023-04-06 18:56:16,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:16,910 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:16,910 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c1504e0a89a4202f82012694b63b21af: 2023-04-06 18:56:16,915 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:56:16,929 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680807376918"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807376918"}]},"ts":"1680807376918"} 2023-04-06 18:56:16,953 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:56:16,955 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:56:16,959 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807376955"}]},"ts":"1680807376955"} 2023-04-06 18:56:16,964 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-06 18:56:16,970 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c1504e0a89a4202f82012694b63b21af, ASSIGN}] 2023-04-06 18:56:16,973 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c1504e0a89a4202f82012694b63b21af, ASSIGN 2023-04-06 18:56:16,974 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c1504e0a89a4202f82012694b63b21af, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:56:17,125 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c1504e0a89a4202f82012694b63b21af, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:17,126 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680807377125"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807377125"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807377125"}]},"ts":"1680807377125"} 2023-04-06 18:56:17,130 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c1504e0a89a4202f82012694b63b21af, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:17,290 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:17,291 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c1504e0a89a4202f82012694b63b21af, NAME => 'hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:17,292 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:17,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,296 INFO [StoreOpener-c1504e0a89a4202f82012694b63b21af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,298 DEBUG [StoreOpener-c1504e0a89a4202f82012694b63b21af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/info 2023-04-06 18:56:17,298 DEBUG [StoreOpener-c1504e0a89a4202f82012694b63b21af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/info 2023-04-06 18:56:17,299 INFO [StoreOpener-c1504e0a89a4202f82012694b63b21af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c1504e0a89a4202f82012694b63b21af columnFamilyName info 2023-04-06 18:56:17,299 INFO [StoreOpener-c1504e0a89a4202f82012694b63b21af-1] regionserver.HStore(310): Store=c1504e0a89a4202f82012694b63b21af/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:17,301 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,303 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c1504e0a89a4202f82012694b63b21af 2023-04-06 18:56:17,311 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:56:17,312 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c1504e0a89a4202f82012694b63b21af; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73926561, jitterRate=0.10159160196781158}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 18:56:17,312 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c1504e0a89a4202f82012694b63b21af: 2023-04-06 18:56:17,314 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af., pid=6, masterSystemTime=1680807377284 2023-04-06 18:56:17,318 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:17,318 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:56:17,320 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c1504e0a89a4202f82012694b63b21af, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:17,320 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680807377319"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807377319"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807377319"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807377319"}]},"ts":"1680807377319"} 2023-04-06 18:56:17,328 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-06 18:56:17,328 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c1504e0a89a4202f82012694b63b21af, server=jenkins-hbase20.apache.org,34607,1680807373848 in 194 msec 2023-04-06 18:56:17,332 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-06 18:56:17,333 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c1504e0a89a4202f82012694b63b21af, ASSIGN in 358 msec 2023-04-06 18:56:17,334 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:56:17,335 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807377334"}]},"ts":"1680807377334"} 2023-04-06 18:56:17,337 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-06 18:56:17,340 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:56:17,343 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3080 sec 2023-04-06 18:56:17,345 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-06 18:56:17,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-06 18:56:17,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:56:17,396 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-06 18:56:17,414 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 18:56:17,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-06 18:56:17,431 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-06 18:56:17,443 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 18:56:17,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-06 18:56:17,461 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-06 18:56:17,462 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-06 18:56:17,463 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.542sec 2023-04-06 18:56:17,464 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-06 18:56:17,466 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-06 18:56:17,466 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-06 18:56:17,467 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45203,1680807372735-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-06 18:56:17,467 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,45203,1680807372735-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-06 18:56:17,477 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-06 18:56:17,487 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x1545112e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:17,491 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d85639, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:17,506 DEBUG [hconnection-0x3192c45d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:17,516 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:17,525 INFO [Listener at localhost.localdomain/41219] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:56:17,546 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=166, AvailableMemoryMB=3531 2023-04-06 18:56:17,560 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:56:17,564 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:56:17,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:56:17,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:17,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:17,579 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:56:17,581 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:56:17,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-06 18:56:17,584 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:17,586 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 empty. 2023-04-06 18:56:17,586 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:17,586 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:56:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 18:56:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 18:56:17,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 18:56:18,006 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:18,008 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 97c1cc7a8c861764bde589e4ad2af6a8, NAME => 'TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 97c1cc7a8c861764bde589e4ad2af6a8, disabling compactions & flushes 2023-04-06 18:56:18,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. after waiting 0 ms 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:18,032 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:56:18,035 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807378034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807378034"}]},"ts":"1680807378034"} 2023-04-06 18:56:18,038 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:56:18,040 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:56:18,040 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807378040"}]},"ts":"1680807378040"} 2023-04-06 18:56:18,044 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:56:18,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, ASSIGN}] 2023-04-06 18:56:18,050 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, ASSIGN 2023-04-06 18:56:18,053 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:56:18,205 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=97c1cc7a8c861764bde589e4ad2af6a8, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:18,205 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807378205"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807378205"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807378205"}]},"ts":"1680807378205"} 2023-04-06 18:56:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 18:56:18,208 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:18,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 97c1cc7a8c861764bde589e4ad2af6a8, NAME => 'TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,372 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,374 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A 2023-04-06 18:56:18,374 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A 2023-04-06 18:56:18,376 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:18,376 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97c1cc7a8c861764bde589e4ad2af6a8 columnFamilyName A 2023-04-06 18:56:18,378 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(310): Store=97c1cc7a8c861764bde589e4ad2af6a8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:18,378 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,380 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B 2023-04-06 18:56:18,380 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B 2023-04-06 18:56:18,381 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:18,381 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97c1cc7a8c861764bde589e4ad2af6a8 columnFamilyName B 2023-04-06 18:56:18,382 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(310): Store=97c1cc7a8c861764bde589e4ad2af6a8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:18,382 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,384 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C 2023-04-06 18:56:18,384 DEBUG [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C 2023-04-06 18:56:18,385 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:18,385 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 97c1cc7a8c861764bde589e4ad2af6a8 columnFamilyName C 2023-04-06 18:56:18,386 INFO [StoreOpener-97c1cc7a8c861764bde589e4ad2af6a8-1] regionserver.HStore(310): Store=97c1cc7a8c861764bde589e4ad2af6a8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:18,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:56:18,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:18,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:56:18,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 97c1cc7a8c861764bde589e4ad2af6a8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68793757, jitterRate=0.02510686218738556}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:56:18,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:18,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., pid=11, masterSystemTime=1680807378361 2023-04-06 18:56:18,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:18,408 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=97c1cc7a8c861764bde589e4ad2af6a8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:18,408 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807378407"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807378407"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807378407"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807378407"}]},"ts":"1680807378407"} 2023-04-06 18:56:18,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-06 18:56:18,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 in 203 msec 2023-04-06 18:56:18,417 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-06 18:56:18,418 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, ASSIGN in 367 msec 2023-04-06 18:56:18,419 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:56:18,419 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807378419"}]},"ts":"1680807378419"} 2023-04-06 18:56:18,422 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:56:18,425 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:56:18,428 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 850 msec 2023-04-06 18:56:18,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 18:56:18,709 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-06 18:56:18,713 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x490ff358 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,717 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ad5f945, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,720 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,724 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,727 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:56:18,731 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:56:18,741 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,748 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142c8aea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,749 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x45864842 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,762 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,762 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,776 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f537ba4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,777 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x66ca212c to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,786 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46b5e40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,787 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x1ecabb06 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,794 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@127a87ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,795 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,803 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,803 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x3e044e22 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,845 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a7ef79d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,847 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6ab9467e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,853 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@776686ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,854 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x18f7ec13 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:18,861 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@385dd3c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:18,878 DEBUG [hconnection-0x6ff5cb68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,878 DEBUG [hconnection-0x3c8bee42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,882 DEBUG [hconnection-0x35d90a24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,882 DEBUG [hconnection-0x53790481-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,883 DEBUG [hconnection-0x7328ed9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,892 DEBUG [hconnection-0x4f512920-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,892 DEBUG [hconnection-0x8ad2eae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,896 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,897 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,906 DEBUG [hconnection-0x719f33c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,908 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,912 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,913 DEBUG [hconnection-0x32f57afc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:18,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:18,936 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,950 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52672, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,950 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,950 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,963 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:18,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:19,010 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:19,010 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:19,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:19,017 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,017 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,021 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:19,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:19,040 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:19,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:19,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:19,040 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:56:19,110 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:19,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:19,120 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:19,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:19,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:19,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:19,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:19,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:19,138 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,171 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:19,171 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:19,172 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:19,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:19,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,178 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:19,178 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:19,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,183 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:19,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3727be3c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:19,203 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:19,203 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:19,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:19,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:19,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:19,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:19,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:19,272 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:19,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:19,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:19,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:19,272 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:19,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:19,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807439262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807439270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807439275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807439275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807439282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,330 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:19,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:19,330 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,330 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,333 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:19,333 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:19,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f226f2dae102401783a5540eafbc8102 2023-04-06 18:56:19,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807439387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807439390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807439391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807439387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807439390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:19,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:19,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:19,471 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:19,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:19,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:19,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:19,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:19,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:19,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:19,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:19,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:19,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:19,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:19,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:19,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:19,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:19,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:19,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/cfdcb49a86254443a3947ffc609f237e 2023-04-06 18:56:19,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,513 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:19,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d62af8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:19,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:19,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,525 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,526 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:19,526 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:19,526 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:19,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:19,543 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:19,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:19,550 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:19,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:19,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:19,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:19,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:19,550 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:19,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:19,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:19,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:19,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:19,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:19,577 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807439597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807439600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807439601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807439615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:19,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:19,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:19,692 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:19,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:19,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:19,696 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,696 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:19,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:19,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:19,699 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:19,699 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:19,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:19,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:19,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:19,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:19,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:19,713 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45623fe2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:19,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:19,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,717 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:19,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:19,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:19,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:19,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:19,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:19,727 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:19,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:19,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:19,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:19,728 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:19,730 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,730 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,747 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,748 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:19,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,748 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,749 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:19,748 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,749 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:19,849 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:19,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:19,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:19,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:19,871 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:19,872 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:19,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:19,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:19,884 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:19,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:19,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:19,887 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,887 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:19,887 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:19,887 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:19,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:19,894 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:19,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386d1a01[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:19,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:19,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807439905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807439906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807439908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,909 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807439908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:19,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,910 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:19,910 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:19,910 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:19,920 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:19,920 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,920 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807439923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:19,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:19,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:19,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:19,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:19,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:19,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:19,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:19,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:19,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:19,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:19,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:19,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:19,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:19,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/46290291ddfc4c1589fecc3a6db07e77 2023-04-06 18:56:20,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f226f2dae102401783a5540eafbc8102 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102 2023-04-06 18:56:20,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102, entries=250, sequenceid=13, filesize=16.2 K 2023-04-06 18:56:20,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/cfdcb49a86254443a3947ffc609f237e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e 2023-04-06 18:56:20,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:20,093 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:20,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:20,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 18:56:20,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:20,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:20,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:20,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:20,111 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:20,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:20,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:20,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:20,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:20,131 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:20,131 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:20,132 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:20,132 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:20,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:20,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:20,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:20,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:20,139 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:20,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:20,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3231956e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:20,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:20,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,146 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/46290291ddfc4c1589fecc3a6db07e77 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77 2023-04-06 18:56:20,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:20,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:20,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:20,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:20,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:20,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:20,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:20,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:20,163 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:20,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 18:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 97c1cc7a8c861764bde589e4ad2af6a8 in 1095ms, sequenceid=13, compaction requested=false 2023-04-06 18:56:20,179 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 18:56:20,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:20,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:20,187 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:20,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:20,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:20,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:20,187 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:20,189 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,189 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,189 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:20,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:20,190 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:20,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:20,202 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:20,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:20,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:20,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:20,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:20,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,221 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:20,221 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:20,221 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:20,221 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:20,321 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:20,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:20,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:20,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:20,370 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:20,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:20,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:20,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:20,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:20,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:20,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:20,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:20,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:20,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:20,410 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:20,410 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:20,430 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:20,431 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:20,431 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:20,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:20,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:20,434 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=114.05 KB heapSize=339.64 KB 2023-04-06 18:56:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:20,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:20,459 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:20,461 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:20,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807440479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807440479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807440486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807440487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807440493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,520 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f576d36cc5da440aa5650f31ae158837 2023-04-06 18:56:20,572 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/7d897bafb38647ad9b8a1f0c6642a115 2023-04-06 18:56:20,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807440595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807440596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807440602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807440602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807440614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807440805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807440808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807440811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807440810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:20,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807440819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,051 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/86c8f052d5ab4e659d3665cc3bc8a893 2023-04-06 18:56:21,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f576d36cc5da440aa5650f31ae158837 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837 2023-04-06 18:56:21,086 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 18:56:21,089 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/7d897bafb38647ad9b8a1f0c6642a115 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115 2023-04-06 18:56:21,108 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 18:56:21,117 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/86c8f052d5ab4e659d3665cc3bc8a893 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893 2023-04-06 18:56:21,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807441117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807441122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807441122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,133 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 18:56:21,137 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~399.84 KB/409440, currentSize=40.25 KB/41220 for 97c1cc7a8c861764bde589e4ad2af6a8 in 702ms, sequenceid=36, compaction requested=false 2023-04-06 18:56:21,138 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:21,138 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:21,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:21,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:21,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:21,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:21,142 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:21,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:21,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:21,142 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:21,142 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:21,143 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:21,143 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:21,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e1e83a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:21,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:21,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:21,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:21,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:21,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:21,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:21,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:21,177 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:21,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:21,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:21,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:21,178 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:21,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:21,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:21,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:21,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:21,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807441241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807441241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:21,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:21,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:21,305 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:21,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:21,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:21,307 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,307 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:21,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:21,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:21,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:21,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:21,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:21,311 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,311 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:21,311 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:21,311 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:21,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,313 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:21,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a8468b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:21,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:21,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:21,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:21,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:21,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:21,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:21,341 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:21,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:21,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:21,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:21,341 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:21,350 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,350 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,350 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,351 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807441348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807441349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,360 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:56:21,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:21,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,361 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:21,362 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:21,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,462 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:21,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:21,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:21,471 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:21,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:21,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:21,473 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,473 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,474 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,474 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:21,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:21,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:21,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:21,478 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:21,478 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:21,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:21,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:21,486 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:21,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6745ef6d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:21,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:21,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,497 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-06 18:56:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:21,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:21,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:21,502 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-06 18:56:21,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:21,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:21,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:21,520 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:21,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:21,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:21,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:21,520 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:21,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,523 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,523 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,542 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,543 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,543 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:21,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,550 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:21,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,550 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:21,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807441558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807441557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/78f7d7c43ef54bdf86e3528c2504ae7f 2023-04-06 18:56:21,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807441642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:21,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807441647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:21,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807441649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:21,659 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:21,659 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:21,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:21,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/1e23c30f4d8e485c9d54a4c9e2d2de7d 2023-04-06 18:56:21,670 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:21,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:21,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:21,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:21,690 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:21,690 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:21,690 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,690 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:21,690 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:21,690 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:21,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:21,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:21,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:21,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e19c0e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:21,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:21,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,744 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:21,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:21,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:21,746 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,747 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:21,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/a841390e4cdb4ef4be7303363084ad6a 2023-04-06 18:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:21,761 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:21,761 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:21,761 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:21,761 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:21,761 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:21,764 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,764 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,764 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:21,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,767 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,775 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/78f7d7c43ef54bdf86e3528c2504ae7f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f 2023-04-06 18:56:21,785 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,785 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:21,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,786 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:21,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:21,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:21,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:21,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:21,793 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:21,798 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f, entries=200, sequenceid=47, filesize=13.9 K 2023-04-06 18:56:21,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/1e23c30f4d8e485c9d54a4c9e2d2de7d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d 2023-04-06 18:56:21,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d, entries=100, sequenceid=47, filesize=9.3 K 2023-04-06 18:56:21,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/a841390e4cdb4ef4be7303363084ad6a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a 2023-04-06 18:56:21,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a, entries=100, sequenceid=47, filesize=9.3 K 2023-04-06 18:56:21,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 97c1cc7a8c861764bde589e4ad2af6a8 in 705ms, sequenceid=47, compaction requested=true 2023-04-06 18:56:21,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:21,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:21,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:21,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:21,850 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:21,852 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:21,856 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42674 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:21,856 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:21,859 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:21,859 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:21,859 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,859 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:21,859 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=32.5 K 2023-04-06 18:56:21,859 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=41.7 K 2023-04-06 18:56:21,862 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting cfdcb49a86254443a3947ffc609f237e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807378998 2023-04-06 18:56:21,864 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f226f2dae102401783a5540eafbc8102, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807378998 2023-04-06 18:56:21,869 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7d897bafb38647ad9b8a1f0c6642a115, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680807379267 2023-04-06 18:56:21,871 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f576d36cc5da440aa5650f31ae158837, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680807379267 2023-04-06 18:56:21,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:21,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=134.18 KB heapSize=399.89 KB 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:21,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:21,878 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1e23c30f4d8e485c9d54a4c9e2d2de7d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807381136 2023-04-06 18:56:21,878 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 78f7d7c43ef54bdf86e3528c2504ae7f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807380478 2023-04-06 18:56:21,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:21,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:21,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:21,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:21,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:21,942 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:21,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:21,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/077fc4a03ac44313b1649934eaaac67c 2023-04-06 18:56:21,950 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#11 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:21,950 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:21,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:21,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:21,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:21,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:21,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:21,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:21,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:21,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807441979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:21,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807441979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:21,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:21,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:21,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:21,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:22,004 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,004 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:22,004 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:22,005 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:22,006 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,006 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ad0f30[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:22,009 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:22,009 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,010 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:22,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:22,012 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:22,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:22,016 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:22,016 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:22,016 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:22,016 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:22,016 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:22,028 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,028 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,028 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,034 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,034 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,034 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,034 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:22,035 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:22,035 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:22,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/da7c8f20e43e48babe1665e4f879489c 2023-04-06 18:56:22,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807442086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807442108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:22,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:22,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:22,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:22,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:22,145 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:22,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:22,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:22,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:22,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:22,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:22,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:22,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:22,193 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,193 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:22,193 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:22,193 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:22,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:22,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40449b10[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:22,203 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:22,203 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,204 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:22,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:22,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:22,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:22,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:22,223 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:22,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:22,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:22,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:22,223 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:22,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,227 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:22,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:22,235 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:22,235 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:22,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807442296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807442319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,335 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:22,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:22,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:22,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:22,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:22,354 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:22,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:22,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:22,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:22,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:22,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:22,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:22,386 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:22,386 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:22,386 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:22,386 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:22,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:22,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:22,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8323af2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:22,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:22,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:22,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:22,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:22,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:22,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,397 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:22,407 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:22,407 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:22,407 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:22,407 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:22,407 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:22,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,423 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,427 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:22,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:22,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,435 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:22,435 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:22,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,465 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/7b050d285bf3427d9fb2652097dcbcd2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/7b050d285bf3427d9fb2652097dcbcd2 2023-04-06 18:56:22,465 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/c3817c423a2a48a7ad927187529a0d3a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c3817c423a2a48a7ad927187529a0d3a 2023-04-06 18:56:22,488 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 7b050d285bf3427d9fb2652097dcbcd2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:22,488 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into c3817c423a2a48a7ad927187529a0d3a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:22,490 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:22,490 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=13, startTime=1680807381850; duration=0sec 2023-04-06 18:56:22,491 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:22,491 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=13, startTime=1680807381848; duration=0sec 2023-04-06 18:56:22,492 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:22,492 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:22,494 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:22,494 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:22,494 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:22,494 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,495 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=32.5 K 2023-04-06 18:56:22,495 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 46290291ddfc4c1589fecc3a6db07e77, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807378998 2023-04-06 18:56:22,496 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 86c8f052d5ab4e659d3665cc3bc8a893, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680807379267 2023-04-06 18:56:22,498 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a841390e4cdb4ef4be7303363084ad6a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807381136 2023-04-06 18:56:22,534 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#14 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:22,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:22,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:22,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:22,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:22,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:22,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:22,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:22,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f34d57524b6647eabbaa57db59bba309 2023-04-06 18:56:22,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:22,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:22,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:22,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:22,582 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:22,582 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:22,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:22,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:22,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:22,587 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:22,587 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:22,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:22,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:22,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:22,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/077fc4a03ac44313b1649934eaaac67c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c 2023-04-06 18:56:22,590 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/7dd955a6cd8b43639d4479db5c8138da as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7dd955a6cd8b43639d4479db5c8138da 2023-04-06 18:56:22,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d66c616[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:22,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:22,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807442601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,603 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:22,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:22,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:22,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:22,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c, entries=200, sequenceid=70, filesize=13.9 K 2023-04-06 18:56:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,616 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 7dd955a6cd8b43639d4479db5c8138da(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:22,616 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:22,616 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=13, startTime=1680807381850; duration=0sec 2023-04-06 18:56:22,617 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:22,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/da7c8f20e43e48babe1665e4f879489c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c 2023-04-06 18:56:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:22,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:22,626 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:22,626 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:22,626 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:22,626 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:22,626 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:22,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:22,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,641 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c, entries=150, sequenceid=70, filesize=11.6 K 2023-04-06 18:56:22,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f34d57524b6647eabbaa57db59bba309 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309 2023-04-06 18:56:22,652 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,652 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:22,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:22,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:22,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:22,652 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:22,652 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:22,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:22,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:22,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:22,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:22,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807442639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807442664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807442665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807442666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309, entries=150, sequenceid=70, filesize=11.6 K 2023-04-06 18:56:22,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~399.84 KB/409440, currentSize=46.96 KB/48090 for 97c1cc7a8c861764bde589e4ad2af6a8 in 810ms, sequenceid=70, compaction requested=false 2023-04-06 18:56:22,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:22,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:22,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:22,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:22,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:22,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:22,760 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:22,760 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:22,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:22,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:22,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:22,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:22,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:22,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:22,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:22,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:22,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:22,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:22,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:22,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:22,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:22,767 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:22,828 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/92c19baa35f04c9e9ceb1b35169bb385 2023-04-06 18:56:22,891 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 18:56:22,892 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-06 18:56:23,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:23,110 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:23,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807383166, max timestamp=1680807383179Num uniques 859; 2023-04-06 18:56:23,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:23,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807383171, max timestamp=1680807383179Num uniques 859; 2023-04-06 18:56:23,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:23,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807383171, max timestamp=1680807383183Num uniques 859; 2023-04-06 18:56:23,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:23,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807443251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807443256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,271 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/9ab829cb0c3a4fa387d48f861b5b043b 2023-04-06 18:56:23,323 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/afa928f0db3a449691fff30847e7ea63 2023-04-06 18:56:23,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/92c19baa35f04c9e9ceb1b35169bb385 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385 2023-04-06 18:56:23,346 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 18:56:23,353 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/9ab829cb0c3a4fa387d48f861b5b043b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b 2023-04-06 18:56:23,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807443358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,368 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 18:56:23,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807443364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,372 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/afa928f0db3a449691fff30847e7ea63 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63 2023-04-06 18:56:23,396 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 18:56:23,398 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 97c1cc7a8c861764bde589e4ad2af6a8 in 631ms, sequenceid=83, compaction requested=true 2023-04-06 18:56:23,398 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:23,399 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:23,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:23,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:23,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:23,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:23,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:23,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8605[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:23,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:23,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,405 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,406 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,406 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:23,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:23,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:23,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:23,413 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:23,413 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:23,413 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:23,414 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:23,414 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:23,415 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,415 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:23,415 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:23,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:23,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:23,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:23,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:23,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:23,429 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:23,429 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:23,529 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:23,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:23,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:23,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:23,534 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:23,534 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:23,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:23,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:23,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:23,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,536 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:23,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:23,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:23,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:23,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:23,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:23,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:23,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:23,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:23,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:23,549 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:23,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:23,549 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:23,550 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-04-06 18:56:23,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:23,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:23,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:23,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:23,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:23,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:23,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807443592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807443594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807443697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807443697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807443899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:23,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807443901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:23,975 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/0dcb22003ee84b839b36b2669defcab5 2023-04-06 18:56:24,016 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/c66e76b824c049e0b9331ace7cef1775 2023-04-06 18:56:24,064 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/9c0f2380ad98429e9e567b82de9b775e 2023-04-06 18:56:24,076 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/0dcb22003ee84b839b36b2669defcab5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5 2023-04-06 18:56:24,086 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5, entries=150, sequenceid=120, filesize=11.6 K 2023-04-06 18:56:24,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/c66e76b824c049e0b9331ace7cef1775 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775 2023-04-06 18:56:24,101 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775, entries=150, sequenceid=120, filesize=11.6 K 2023-04-06 18:56:24,110 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/9c0f2380ad98429e9e567b82de9b775e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e 2023-04-06 18:56:24,121 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e, entries=150, sequenceid=120, filesize=11.6 K 2023-04-06 18:56:24,127 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=46.96 KB/48090 for 97c1cc7a8c861764bde589e4ad2af6a8 in 573ms, sequenceid=120, compaction requested=true 2023-04-06 18:56:24,127 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,127 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,132 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64170103[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,156 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,156 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,170 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,170 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,171 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,175 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:24,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:24,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:24,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a2c10beb6d6a49f7b14feff2439dba6a 2023-04-06 18:56:24,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:24,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:24,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/e16449e9cc2e4b51a3aa0aa19d0134a7 2023-04-06 18:56:24,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:24,290 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:24,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:24,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:24,292 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,292 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,293 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,293 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:24,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:24,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:24,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:24,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:24,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:24,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:24,302 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,303 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:24,303 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:24,303 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,313 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7deb52d9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,317 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,321 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,321 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807444328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807444329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,335 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,335 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/1d6528c5826c487eac5401af7a86c8c8 2023-04-06 18:56:24,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a2c10beb6d6a49f7b14feff2439dba6a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a 2023-04-06 18:56:24,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a, entries=150, sequenceid=132, filesize=11.8 K 2023-04-06 18:56:24,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/e16449e9cc2e4b51a3aa0aa19d0134a7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7 2023-04-06 18:56:24,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7, entries=150, sequenceid=132, filesize=11.8 K 2023-04-06 18:56:24,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/1d6528c5826c487eac5401af7a86c8c8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8 2023-04-06 18:56:24,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8, entries=150, sequenceid=132, filesize=11.8 K 2023-04-06 18:56:24,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 97c1cc7a8c861764bde589e4ad2af6a8 in 187ms, sequenceid=132, compaction requested=true 2023-04-06 18:56:24,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:24,398 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:24,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:24,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:24,401 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:24,404 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62023 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:24,405 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:24,405 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,405 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/7b050d285bf3427d9fb2652097dcbcd2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=60.6 K 2023-04-06 18:56:24,406 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59683 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:24,406 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:24,406 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,406 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c3817c423a2a48a7ad927187529a0d3a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=58.3 K 2023-04-06 18:56:24,406 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7b050d285bf3427d9fb2652097dcbcd2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807380421 2023-04-06 18:56:24,408 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c3817c423a2a48a7ad927187529a0d3a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807380421 2023-04-06 18:56:24,409 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting da7c8f20e43e48babe1665e4f879489c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807381226 2023-04-06 18:56:24,409 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 077fc4a03ac44313b1649934eaaac67c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807381226 2023-04-06 18:56:24,410 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 92c19baa35f04c9e9ceb1b35169bb385, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680807381905 2023-04-06 18:56:24,410 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9ab829cb0c3a4fa387d48f861b5b043b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680807381905 2023-04-06 18:56:24,411 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0dcb22003ee84b839b36b2669defcab5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680807383239 2023-04-06 18:56:24,411 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c66e76b824c049e0b9331ace7cef1775, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680807383239 2023-04-06 18:56:24,412 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e16449e9cc2e4b51a3aa0aa19d0134a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:24,413 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a2c10beb6d6a49f7b14feff2439dba6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:24,435 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:24,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:24,444 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:24,447 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:24,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:24,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:24,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-06 18:56:24,451 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:24,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807384312, max timestamp=1680807384441Num uniques 859; 2023-04-06 18:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:24,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:24,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807384312, max timestamp=1680807384441Num uniques 859; 2023-04-06 18:56:24,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:24,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:24,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807384312, max timestamp=1680807384441Num uniques 859; 2023-04-06 18:56:24,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:24,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:24,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,462 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:24,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:24,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:24,463 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,463 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:24,463 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:24,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,463 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,464 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,464 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,476 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,478 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,478 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16566d04[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,484 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,487 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,487 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,498 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,498 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,510 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:24,514 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/4bf005a4bbc24e84b31e6e34bab1119f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4bf005a4bbc24e84b31e6e34bab1119f 2023-04-06 18:56:24,516 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/e3ba2056bcd4481a840e9837b97e970a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e3ba2056bcd4481a840e9837b97e970a 2023-04-06 18:56:24,536 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 4bf005a4bbc24e84b31e6e34bab1119f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:24,539 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,539 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=11, startTime=1680807384398; duration=0sec 2023-04-06 18:56:24,537 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into e3ba2056bcd4481a840e9837b97e970a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:24,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f305cba27b56488f8f3a0a486d21e521 2023-04-06 18:56:24,541 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,541 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=11, startTime=1680807384398; duration=0sec 2023-04-06 18:56:24,541 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:24,541 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:24,539 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:24,545 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59683 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:24,546 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:24,546 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,546 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7dd955a6cd8b43639d4479db5c8138da, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=58.3 K 2023-04-06 18:56:24,547 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7dd955a6cd8b43639d4479db5c8138da, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807380421 2023-04-06 18:56:24,551 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f34d57524b6647eabbaa57db59bba309, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807381226 2023-04-06 18:56:24,552 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting afa928f0db3a449691fff30847e7ea63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680807381905 2023-04-06 18:56:24,553 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9c0f2380ad98429e9e567b82de9b775e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680807383239 2023-04-06 18:56:24,554 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1d6528c5826c487eac5401af7a86c8c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:24,563 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807384528, max timestamp=1680807384559Num uniques 859; 2023-04-06 18:56:24,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807384528, max timestamp=1680807384559Num uniques 859; 2023-04-06 18:56:24,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:24,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:24,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807384528, max timestamp=1680807384559Num uniques 859; 2023-04-06 18:56:24,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:24,596 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:24,610 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:24,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:24,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:24,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/a5be464d748043ca9804cd7062c06de1 2023-04-06 18:56:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:24,630 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:24,630 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,631 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:24,631 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:24,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807384602, max timestamp=1680807384628Num uniques 1068; 2023-04-06 18:56:24,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-06 18:56:24,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807384602, max timestamp=1680807384628Num uniques 1068; 2023-04-06 18:56:24,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-06 18:56:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807384602, max timestamp=1680807384628Num uniques 1068; 2023-04-06 18:56:24,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-06 18:56:24,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:24,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:24,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:24,642 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:24,642 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:24,647 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,647 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:24,647 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:24,647 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,657 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/c090e1d7be66470db8eb8161ce2cd5f6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/c090e1d7be66470db8eb8161ce2cd5f6 2023-04-06 18:56:24,658 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,659 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,664 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,664 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a759e65[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,672 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,672 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,675 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into c090e1d7be66470db8eb8161ce2cd5f6(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:24,675 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,675 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=11, startTime=1680807384398; duration=0sec 2023-04-06 18:56:24,676 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:24,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,676 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/66b50a3783654107b30f8de78109769f 2023-04-06 18:56:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,678 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/f305cba27b56488f8f3a0a486d21e521 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521 2023-04-06 18:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,693 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,693 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,696 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,698 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,698 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:24,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521, entries=250, sequenceid=154, filesize=16.4 K 2023-04-06 18:56:24,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/a5be464d748043ca9804cd7062c06de1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1 2023-04-06 18:56:24,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1, entries=150, sequenceid=154, filesize=11.8 K 2023-04-06 18:56:24,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/66b50a3783654107b30f8de78109769f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f 2023-04-06 18:56:24,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f, entries=150, sequenceid=154, filesize=11.8 K 2023-04-06 18:56:24,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=93.93 KB/96180 for 97c1cc7a8c861764bde589e4ad2af6a8 in 273ms, sequenceid=154, compaction requested=false 2023-04-06 18:56:24,723 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:24,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=114.05 KB heapSize=315 KB 2023-04-06 18:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:24,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:24,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807444757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807444759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807444761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807444763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807444764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:24,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:24,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:24,804 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:24,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:24,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:24,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:24,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:24,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:24,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:24,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:24,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:24,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:24,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36369531[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,842 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,842 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,852 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,852 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807444865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807444865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807444866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807444867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807444867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,954 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:24,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:24,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:24,958 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:24,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:24,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:24,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:24,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:24,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:24,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:24,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:24,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:24,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:24,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:24,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:24,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:24,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:24,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:24,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:24,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11ec74b4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:24,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:24,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:24,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:24,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:24,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:24,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:24,971 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:24,974 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:24,974 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:24,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:24,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:24,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:24,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:24,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:24,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:24,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:24,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:24,997 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:24,997 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:24,997 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:24,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:24,997 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:24,997 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:24,997 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807445070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807445071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807445071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807445072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807445072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,097 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,101 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:25,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@347715c0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,114 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,114 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,116 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,127 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,126 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:25,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d754bf7989ac41c1954d590cdf911b94 2023-04-06 18:56:25,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/eab1a8ccf9ed4ccd8153af479e4600e6 2023-04-06 18:56:25,227 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,235 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,237 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,242 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,242 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,242 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,242 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7237239d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,248 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,252 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,252 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,253 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,253 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,253 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,260 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,260 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:25,360 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,364 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,365 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,370 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,370 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,370 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,370 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,370 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,377 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807445375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52672 deadline: 1680807445375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807445376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807445377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702abff7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807445380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,387 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,390 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,390 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,399 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:25,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,399 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,399 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,502 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2912c92[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,510 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,510 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,517 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,517 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/8d6a92c91ca24c72b0f836319a2c70ef 2023-04-06 18:56:25,617 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,622 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d754bf7989ac41c1954d590cdf911b94 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94 2023-04-06 18:56:25,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,626 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,626 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,627 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,627 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,627 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,627 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,627 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,627 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94, entries=200, sequenceid=207, filesize=14.1 K 2023-04-06 18:56:25,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/eab1a8ccf9ed4ccd8153af479e4600e6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6 2023-04-06 18:56:25,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6, entries=150, sequenceid=207, filesize=11.8 K 2023-04-06 18:56:25,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,643 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/8d6a92c91ca24c72b0f836319a2c70ef as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef 2023-04-06 18:56:25,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec4f015[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,657 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,657 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,657 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,658 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef, entries=150, sequenceid=207, filesize=11.8 K 2023-04-06 18:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~334.88 KB/342912, currentSize=60.38 KB/61830 for 97c1cc7a8c861764bde589e4ad2af6a8 in 933ms, sequenceid=207, compaction requested=true 2023-04-06 18:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:25,660 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:25,661 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:25,661 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,662 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:25,663 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:25,663 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:25,663 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:25,663 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,663 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,663 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e3ba2056bcd4481a840e9837b97e970a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=35.5 K 2023-04-06 18:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,663 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4bf005a4bbc24e84b31e6e34bab1119f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=42.5 K 2023-04-06 18:56:25,665 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e3ba2056bcd4481a840e9837b97e970a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:25,663 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,666 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4bf005a4bbc24e84b31e6e34bab1119f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,666 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,667 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f305cba27b56488f8f3a0a486d21e521, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680807384312 2023-04-06 18:56:25,667 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a5be464d748043ca9804cd7062c06de1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680807384314 2023-04-06 18:56:25,668 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d754bf7989ac41c1954d590cdf911b94, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680807384691 2023-04-06 18:56:25,668 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting eab1a8ccf9ed4ccd8153af479e4600e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680807384691 2023-04-06 18:56:25,669 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,669 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,670 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,676 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,676 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,677 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,677 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:25,683 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#33 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:25,686 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:25,777 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:25,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:25,781 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:25,781 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,781 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:25,781 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:25,781 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,782 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,783 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,783 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:25,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:25,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:25,784 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:25,784 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:25,784 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:25,784 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:25,784 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:25,784 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:25,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,801 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/2517ac63cb634d7ca74c192a392e1400 2023-04-06 18:56:25,827 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/212540bffc944ceea81c447af3ca52f2 2023-04-06 18:56:25,854 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/304e67780c1a400bbb1b7af36e8413a9 2023-04-06 18:56:25,861 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/2517ac63cb634d7ca74c192a392e1400 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400 2023-04-06 18:56:25,870 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400, entries=150, sequenceid=219, filesize=11.8 K 2023-04-06 18:56:25,876 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/212540bffc944ceea81c447af3ca52f2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2 2023-04-06 18:56:25,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:25,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:25,889 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2, entries=150, sequenceid=219, filesize=11.8 K 2023-04-06 18:56:25,891 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/304e67780c1a400bbb1b7af36e8413a9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9 2023-04-06 18:56:25,901 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9, entries=150, sequenceid=219, filesize=11.8 K 2023-04-06 18:56:25,903 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=73.80 KB/75570 for 97c1cc7a8c861764bde589e4ad2af6a8 in 119ms, sequenceid=219, compaction requested=true 2023-04-06 18:56:25,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:25,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:25,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 18:56:25,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:25,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:25,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:25,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:25,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:25,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:25,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,910 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:25,910 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:25,911 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,911 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a4119b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:25,915 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:25,915 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,916 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:25,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:25,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:25,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:25,918 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:25,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:25,920 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:25,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:25,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:25,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:25,920 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:25,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:25,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,922 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:25,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:25,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:25,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:25,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,925 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:25,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:25,925 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:25,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:25,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:25,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52638 deadline: 1680807445952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807445955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807445956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807445957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:25,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:25,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807445958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,025 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,029 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,029 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,029 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,029 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,033 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6054cbdf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,039 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,039 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,043 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,043 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,043 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:26,044 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,044 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807446058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807446059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807446063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807446063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807446063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,107 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/d37595c1789e4bfe88e0911de27bc2fb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d37595c1789e4bfe88e0911de27bc2fb 2023-04-06 18:56:26,109 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/98eb426534fc4996bb015e781f25cbdc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/98eb426534fc4996bb015e781f25cbdc 2023-04-06 18:56:26,118 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into d37595c1789e4bfe88e0911de27bc2fb(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:26,118 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:26,118 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=13, startTime=1680807385660; duration=0sec 2023-04-06 18:56:26,118 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:26,118 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:26,119 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 98eb426534fc4996bb015e781f25cbdc(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:26,119 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:26,120 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=13, startTime=1680807385660; duration=0sec 2023-04-06 18:56:26,120 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:26,120 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:26,120 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:26,121 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,121 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/c090e1d7be66470db8eb8161ce2cd5f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=47.3 K 2023-04-06 18:56:26,121 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c090e1d7be66470db8eb8161ce2cd5f6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680807383592 2023-04-06 18:56:26,122 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 66b50a3783654107b30f8de78109769f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1680807384314 2023-04-06 18:56:26,122 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8d6a92c91ca24c72b0f836319a2c70ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680807384691 2023-04-06 18:56:26,123 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 304e67780c1a400bbb1b7af36e8413a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680807384754 2023-04-06 18:56:26,134 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:26,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,147 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,151 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,151 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,151 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,151 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6baa8efb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,160 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,160 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807446261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807446261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807446265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807446265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807446266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,273 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,276 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,276 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,276 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,276 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c79fc3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,280 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,280 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,280 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,280 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,282 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,283 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,287 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,287 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:26,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,288 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,288 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/66cd44b3d6984a54bed3086fa7cc93ca 2023-04-06 18:56:26,388 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,392 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,394 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,395 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,395 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,395 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278a76c9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,399 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,401 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,402 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,402 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,402 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,408 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:26,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,515 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,515 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,515 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,516 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61464bd4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,519 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,519 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,520 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,522 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,522 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,522 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,523 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,523 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,524 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,528 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:26,546 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/0f3558f56411422ebc52e6f37f44cedd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0f3558f56411422ebc52e6f37f44cedd 2023-04-06 18:56:26,553 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 0f3558f56411422ebc52e6f37f44cedd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:26,553 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:26,553 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=12, startTime=1680807385661; duration=0sec 2023-04-06 18:56:26,553 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:26,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807446566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807446566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807446569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807446571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807446573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,628 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,633 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,634 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,635 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,635 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,635 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,636 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,636 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,636 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,636 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,637 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea4dc51[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,640 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,640 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,640 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,641 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,644 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,644 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,646 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,649 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/31cefa718bb54df7b14168adf30adf5f 2023-04-06 18:56:26,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,758 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,760 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,761 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,792 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,792 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,792 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:26,792 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:26,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:26,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,794 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:26,794 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:26,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/80e59de30d6b4396a15dacb23034b016 2023-04-06 18:56:26,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:26,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e379c1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:26,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:26,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/66cd44b3d6984a54bed3086fa7cc93ca as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca 2023-04-06 18:56:26,810 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:26,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:26,810 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,811 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:26,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:26,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca, entries=400, sequenceid=235, filesize=23.4 K 2023-04-06 18:56:26,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:26,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:26,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/31cefa718bb54df7b14168adf30adf5f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f 2023-04-06 18:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:26,821 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:26,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:26,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:26,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:26,821 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:26,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f, entries=150, sequenceid=235, filesize=11.8 K 2023-04-06 18:56:26,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:26,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/80e59de30d6b4396a15dacb23034b016 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016 2023-04-06 18:56:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,834 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016, entries=150, sequenceid=235, filesize=11.8 K 2023-04-06 18:56:26,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:26,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:26,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:26,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:26,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:26,835 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:26,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:26,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 97c1cc7a8c861764bde589e4ad2af6a8 in 929ms, sequenceid=235, compaction requested=true 2023-04-06 18:56:26,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:26,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:26,836 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:26,836 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:26,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:26,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:26,837 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 48438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:26,837 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:26,837 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:26,837 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,837 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:26,838 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,838 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/98eb426534fc4996bb015e781f25cbdc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=47.3 K 2023-04-06 18:56:26,838 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d37595c1789e4bfe88e0911de27bc2fb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=35.6 K 2023-04-06 18:56:26,838 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 98eb426534fc4996bb015e781f25cbdc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680807384691 2023-04-06 18:56:26,838 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d37595c1789e4bfe88e0911de27bc2fb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680807384691 2023-04-06 18:56:26,839 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2517ac63cb634d7ca74c192a392e1400, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680807384754 2023-04-06 18:56:26,839 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 212540bffc944ceea81c447af3ca52f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680807384754 2023-04-06 18:56:26,839 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 66cd44b3d6984a54bed3086fa7cc93ca, keycount=400, bloomtype=ROW, size=23.4 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680807385892 2023-04-06 18:56:26,840 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 31cefa718bb54df7b14168adf30adf5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680807385901 2023-04-06 18:56:26,847 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:26,849 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:26,866 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/03558ef0f1ea4b88a13a3270085eab32 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/03558ef0f1ea4b88a13a3270085eab32 2023-04-06 18:56:26,872 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 03558ef0f1ea4b88a13a3270085eab32(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:26,872 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:26,872 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=13, startTime=1680807386836; duration=0sec 2023-04-06 18:56:26,872 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:26,872 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 18:56:26,873 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:56:26,873 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:56:26,873 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. because compaction request was cancelled 2023-04-06 18:56:26,935 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:26,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:26,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:26,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:26,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:26,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:26,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:26,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:26,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:26,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:26,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:26,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:26,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:26,943 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:26,943 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:26,943 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:26,946 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:26,946 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:26,946 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:26,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:26,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:26,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:26,961 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a42c30e798de46a6ae8c6b2181541cb4 2023-04-06 18:56:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:27,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:27,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807447082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807447082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807447087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807447088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807447088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807447189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807447189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807447191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807447192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807447192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,269 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/fc480e6c14074b988173e29611517cc1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fc480e6c14074b988173e29611517cc1 2023-04-06 18:56:27,275 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into fc480e6c14074b988173e29611517cc1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:27,275 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:27,275 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=13, startTime=1680807386836; duration=0sec 2023-04-06 18:56:27,275 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:27,373 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/50b56f8a85c649f98318fe4ca00fdecb 2023-04-06 18:56:27,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807447393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807447393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807447394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807447394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807447395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807447697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807447700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807447700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807447700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807447700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,787 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/661847caca294ee5a0691fee30de1de1 2023-04-06 18:56:27,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a42c30e798de46a6ae8c6b2181541cb4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4 2023-04-06 18:56:27,803 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 18:56:27,805 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/50b56f8a85c649f98318fe4ca00fdecb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb 2023-04-06 18:56:27,815 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 18:56:27,816 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/661847caca294ee5a0691fee30de1de1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1 2023-04-06 18:56:27,824 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1, entries=150, sequenceid=256, filesize=11.8 K 2023-04-06 18:56:27,825 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 97c1cc7a8c861764bde589e4ad2af6a8 in 879ms, sequenceid=256, compaction requested=true 2023-04-06 18:56:27,826 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:27,826 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:27,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:27,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:27,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:27,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:27,827 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:27,827 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ab6f2a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:27,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:27,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,830 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:27,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:27,830 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,830 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:27,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:27,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:27,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:27,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:27,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:27,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:27,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:27,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:27,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:27,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:27,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:27,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:27,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:27,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,853 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:27,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:27,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:27,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:27,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:27,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:27,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,856 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:27,856 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:27,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:27,956 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:27,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:27,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:27,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:27,960 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:27,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:27,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:27,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:27,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:27,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:27,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:27,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:27,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:27,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:27,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:27,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:27,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:27,963 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:27,963 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:27,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:28,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:28,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:28,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807448219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807448220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807448221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807448221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807448221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807448326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807448326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807448328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807448331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807448332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,390 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/58be823d6ec84af895933737a92f3ea8 2023-04-06 18:56:28,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807448530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807448530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807448531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807448534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807448535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,808 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f3d0472bedcb4507acbb1bce8b6ebcba 2023-04-06 18:56:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807448834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807448835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807448837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807448838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:28,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807448847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,256 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/3236c9cc7a6e4353b03f792e7349b1cc 2023-04-06 18:56:29,264 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/58be823d6ec84af895933737a92f3ea8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8 2023-04-06 18:56:29,271 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8, entries=150, sequenceid=273, filesize=11.9 K 2023-04-06 18:56:29,273 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f3d0472bedcb4507acbb1bce8b6ebcba as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba 2023-04-06 18:56:29,279 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba, entries=150, sequenceid=273, filesize=11.9 K 2023-04-06 18:56:29,280 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/3236c9cc7a6e4353b03f792e7349b1cc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc 2023-04-06 18:56:29,293 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc, entries=150, sequenceid=273, filesize=11.9 K 2023-04-06 18:56:29,294 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=100.63 KB/103050 for 97c1cc7a8c861764bde589e4ad2af6a8 in 1331ms, sequenceid=273, compaction requested=true 2023-04-06 18:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:29,294 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,294 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:29,294 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:29,294 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:29,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:29,300 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:29,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7608680[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:29,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:29,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:29,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:29,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:29,307 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:29,311 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:29,312 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:29,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,345 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:29,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:29,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:29,345 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:29,345 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:29,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:29,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:29,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807449356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807449358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807449360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807449362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807449362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,445 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:29,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:29,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:29,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:29,454 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:29,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:29,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:29,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:29,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:29,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:29,462 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:29,462 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,462 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:29,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807449464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:29,471 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,471 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:29,471 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:29,471 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:29,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:29,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807449465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807449467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807449467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807449467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,473 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fde9469[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:29,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:29,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,476 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:29,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:29,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:29,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:29,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:29,477 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:29,479 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:29,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:29,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:29,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:29,479 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:29,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:29,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,486 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:29,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,486 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:29,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,586 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:29,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:29,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:29,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:29,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:29,595 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:29,595 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:29,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:29,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,597 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:29,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:29,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:29,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:29,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:29,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:29,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:29,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:29,600 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,600 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:29,600 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:29,601 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:29,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:29,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:29,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:29,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:29,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4448d722[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:29,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:29,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:29,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:29,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:29,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:29,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:29,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:29,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:29,644 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:29,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:29,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:29,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:29,644 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:29,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,664 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807449670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,677 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:29,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:29,677 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:29,677 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:29,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807449675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807449675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:29,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807449676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807449676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d05e5b3ed44c4e84ba212e23107e4f96 2023-04-06 18:56:29,777 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:29,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:29,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:29,781 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:29,781 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:29,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:29,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,783 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,783 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:29,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:29,787 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,787 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:29,787 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:29,787 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:29,788 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,788 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1532be[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:29,791 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:29,791 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,792 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:29,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:29,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:29,793 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:29,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:29,795 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:29,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:29,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:29,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:29,795 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:29,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,797 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:29,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:29,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:29,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:29,800 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:29,800 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:29,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:29,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:29,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/53e1b1dc2f78455b8eebdcb5548016a0 2023-04-06 18:56:29,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f7fba0239cb24054aa8e485657698cf2 2023-04-06 18:56:29,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d05e5b3ed44c4e84ba212e23107e4f96 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96 2023-04-06 18:56:29,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96, entries=200, sequenceid=292, filesize=14.3 K 2023-04-06 18:56:29,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/53e1b1dc2f78455b8eebdcb5548016a0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0 2023-04-06 18:56:29,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0, entries=150, sequenceid=292, filesize=11.9 K 2023-04-06 18:56:29,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f7fba0239cb24054aa8e485657698cf2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2 2023-04-06 18:56:29,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2, entries=150, sequenceid=292, filesize=11.9 K 2023-04-06 18:56:29,885 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 97c1cc7a8c861764bde589e4ad2af6a8 in 540ms, sequenceid=292, compaction requested=true 2023-04-06 18:56:29,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:29,885 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:29,887 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51362 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:29,887 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:29,887 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,888 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/03558ef0f1ea4b88a13a3270085eab32, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=50.2 K 2023-04-06 18:56:29,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:29,888 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 03558ef0f1ea4b88a13a3270085eab32, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680807385901 2023-04-06 18:56:29,889 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:29,889 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a42c30e798de46a6ae8c6b2181541cb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680807385945 2023-04-06 18:56:29,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:29,890 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 58be823d6ec84af895933737a92f3ea8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680807387079 2023-04-06 18:56:29,891 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d05e5b3ed44c4e84ba212e23107e4f96, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:29,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48922 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:29,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:29,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fc480e6c14074b988173e29611517cc1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=47.8 K 2023-04-06 18:56:29,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:29,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting fc480e6c14074b988173e29611517cc1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680807385901 2023-04-06 18:56:29,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 50b56f8a85c649f98318fe4ca00fdecb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680807385945 2023-04-06 18:56:29,894 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f3d0472bedcb4507acbb1bce8b6ebcba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680807387079 2023-04-06 18:56:29,894 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 53e1b1dc2f78455b8eebdcb5548016a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:29,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:29,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:29,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:29,916 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:29,921 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:29,922 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:29,922 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,922 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:29,922 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:29,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:29,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:29,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:29,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:29,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:29,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:29,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:29,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:29,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:29,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:29,931 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:56:29,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:29,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:29,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:29,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:29,939 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/be31f6145b184ebe8ee658e796784af1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/be31f6145b184ebe8ee658e796784af1 2023-04-06 18:56:29,953 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into be31f6145b184ebe8ee658e796784af1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:29,953 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:29,953 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=12, startTime=1680807389888; duration=0sec 2023-04-06 18:56:29,953 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:29,953 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:29,956 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60890 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:29,956 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:29,956 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:29,957 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0f3558f56411422ebc52e6f37f44cedd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=59.5 K 2023-04-06 18:56:29,957 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/22a8b6f242f94b70bc75b05523a834cd 2023-04-06 18:56:29,957 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0f3558f56411422ebc52e6f37f44cedd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680807384754 2023-04-06 18:56:29,958 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 80e59de30d6b4396a15dacb23034b016, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680807385901 2023-04-06 18:56:29,959 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 661847caca294ee5a0691fee30de1de1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1680807385945 2023-04-06 18:56:29,959 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3236c9cc7a6e4353b03f792e7349b1cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1680807387079 2023-04-06 18:56:29,960 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f7fba0239cb24054aa8e485657698cf2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:29,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:29,980 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:30,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,338 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/beb810620a0443aebdf799d222910f96 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/beb810620a0443aebdf799d222910f96 2023-04-06 18:56:30,345 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into beb810620a0443aebdf799d222910f96(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:30,345 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:30,345 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=12, startTime=1680807389885; duration=0sec 2023-04-06 18:56:30,345 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:30,385 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/16610ce8fa6b4f47b41ab1dc7d8bc919 2023-04-06 18:56:30,399 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/bd24f13b199d41dfbbbc37e95dcf9607 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/bd24f13b199d41dfbbbc37e95dcf9607 2023-04-06 18:56:30,399 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/a1c5bf3613ea4d8bb5e653e05905fe12 2023-04-06 18:56:30,406 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/22a8b6f242f94b70bc75b05523a834cd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd 2023-04-06 18:56:30,410 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into bd24f13b199d41dfbbbc37e95dcf9607(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:30,410 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:30,410 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=11, startTime=1680807389892; duration=0sec 2023-04-06 18:56:30,410 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:30,414 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd, entries=150, sequenceid=305, filesize=11.9 K 2023-04-06 18:56:30,415 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/16610ce8fa6b4f47b41ab1dc7d8bc919 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919 2023-04-06 18:56:30,421 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919, entries=150, sequenceid=305, filesize=11.9 K 2023-04-06 18:56:30,423 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/a1c5bf3613ea4d8bb5e653e05905fe12 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12 2023-04-06 18:56:30,435 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12, entries=150, sequenceid=305, filesize=11.9 K 2023-04-06 18:56:30,436 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 97c1cc7a8c861764bde589e4ad2af6a8 in 505ms, sequenceid=305, compaction requested=false 2023-04-06 18:56:30,436 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:30,436 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:30,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:30,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:30,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:30,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:30,437 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,437 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:30,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:30,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:30,439 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:30,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4842f9da[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:30,439 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,440 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:30,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:30,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:30,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:30,441 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:30,441 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:30,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,441 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:30,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:30,445 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:30,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:30,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:30,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:30,445 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:30,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:30,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:30,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:30,461 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:30,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:30,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:30,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:30,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:30,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:30,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:30,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:30,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:30,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:30,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:30,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:30,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:30,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:30,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:30,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:30,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:30,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:30,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:30,573 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:30,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:30,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:30,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:30,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:30,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:30,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:30,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:30,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:30,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:30,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:30,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:30,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:30,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:30,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:30,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:30,576 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:30,576 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:30,576 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:56:30,576 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:30,576 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:30,577 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:30,577 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:30,577 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:30,577 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:30,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:30,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:30,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807450930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807450933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807450934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807450934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:30,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807450934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:30,992 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/ec749d2eace642c58cc64ba46b3f7032 2023-04-06 18:56:31,032 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/caf414529cc140ca8f26726582e45dcb 2023-04-06 18:56:31,064 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/74881cd8570a4a51a002bac19b080e13 2023-04-06 18:56:31,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/ec749d2eace642c58cc64ba46b3f7032 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032 2023-04-06 18:56:31,079 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032, entries=150, sequenceid=328, filesize=11.9 K 2023-04-06 18:56:31,080 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/caf414529cc140ca8f26726582e45dcb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb 2023-04-06 18:56:31,089 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb, entries=150, sequenceid=328, filesize=11.9 K 2023-04-06 18:56:31,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/74881cd8570a4a51a002bac19b080e13 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13 2023-04-06 18:56:31,124 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13, entries=150, sequenceid=328, filesize=11.9 K 2023-04-06 18:56:31,125 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 97c1cc7a8c861764bde589e4ad2af6a8 in 548ms, sequenceid=328, compaction requested=true 2023-04-06 18:56:31,125 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,125 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:31,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:31,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:31,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:31,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:31,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43344167[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:31,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:31,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,135 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:31,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:31,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:31,140 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:31,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:31,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:31,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:31,140 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:31,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,144 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,148 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:31,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,148 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:31,148 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:31,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:31,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:56:31,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:31,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:31,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:31,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,248 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:31,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:31,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:31,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/caa971277c5a4067baeb0891fe34d409 2023-04-06 18:56:31,261 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:31,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:31,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:31,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:31,267 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:31,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:31,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:31,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:31,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:31,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:31,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:31,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:31,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:31,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:31,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807451271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807451273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807451275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,280 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:31,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c76fe4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:31,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:31,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:31,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:31,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:31,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:31,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:31,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807451279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807451279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:31,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:31,290 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:31,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:31,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:31,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:31,290 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:31,291 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,291 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,291 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,292 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,293 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:31,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:31,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:31,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:31,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:31,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/165dee436b9346b48b8847c9767e5a06 2023-04-06 18:56:31,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/59eac5feea90420cb5c74a9528d3a0be 2023-04-06 18:56:31,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/caa971277c5a4067baeb0891fe34d409 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409 2023-04-06 18:56:31,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409, entries=200, sequenceid=343, filesize=14.3 K 2023-04-06 18:56:31,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/165dee436b9346b48b8847c9767e5a06 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06 2023-04-06 18:56:31,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06, entries=150, sequenceid=343, filesize=11.9 K 2023-04-06 18:56:31,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/59eac5feea90420cb5c74a9528d3a0be as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be 2023-04-06 18:56:31,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807451381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be, entries=150, sequenceid=343, filesize=11.9 K 2023-04-06 18:56:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807451383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 97c1cc7a8c861764bde589e4ad2af6a8 in 150ms, sequenceid=343, compaction requested=true 2023-04-06 18:56:31,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:31,391 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:31,391 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:31,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:31,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:31,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:31,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 18:56:31,393 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:31,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:31,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:31,394 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:31,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:31,396 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51798 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:31,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:31,397 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:31,397 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,397 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/be31f6145b184ebe8ee658e796784af1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=48.2 K 2023-04-06 18:56:31,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:31,397 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,397 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting be31f6145b184ebe8ee658e796784af1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:31,397 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/beb810620a0443aebdf799d222910f96, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=50.6 K 2023-04-06 18:56:31,398 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 16610ce8fa6b4f47b41ab1dc7d8bc919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680807389354 2023-04-06 18:56:31,398 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting beb810620a0443aebdf799d222910f96, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:31,398 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting caf414529cc140ca8f26726582e45dcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680807390001 2023-04-06 18:56:31,399 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 22a8b6f242f94b70bc75b05523a834cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680807389354 2023-04-06 18:56:31,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:31,400 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 165dee436b9346b48b8847c9767e5a06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:31,400 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ec749d2eace642c58cc64ba46b3f7032, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680807390001 2023-04-06 18:56:31,401 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting caa971277c5a4067baeb0891fe34d409, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:31,424 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:31,424 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,424 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:31,424 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:31,424 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,424 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:31,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:31,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,437 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:31,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807451427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807451428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807451437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,460 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:31,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/95954368e2cc4665863caa51fa1a0376 2023-04-06 18:56:31,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:31,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:31,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:31,516 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:31,516 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,516 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:31,517 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:31,517 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:31,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:31,522 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,523 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,524 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/85919e12f70a4b9488572c36e1a8cad4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/85919e12f70a4b9488572c36e1a8cad4 2023-04-06 18:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/813d2eacd1654240bf536b6d1c6b4e58 2023-04-06 18:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232535a0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:31,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:31,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,530 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/adf06638e87442c18df2ee542892dde5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/adf06638e87442c18df2ee542892dde5 2023-04-06 18:56:31,532 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into 85919e12f70a4b9488572c36e1a8cad4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:31,532 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,532 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=12, startTime=1680807391391; duration=0sec 2023-04-06 18:56:31,532 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:31,533 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:31,539 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:31,539 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:31,540 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,540 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/bd24f13b199d41dfbbbc37e95dcf9607, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=48.2 K 2023-04-06 18:56:31,541 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting bd24f13b199d41dfbbbc37e95dcf9607, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680807388210 2023-04-06 18:56:31,542 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a1c5bf3613ea4d8bb5e653e05905fe12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680807389354 2023-04-06 18:56:31,542 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 74881cd8570a4a51a002bac19b080e13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680807390001 2023-04-06 18:56:31,542 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:31,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:31,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:31,543 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 59eac5feea90420cb5c74a9528d3a0be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:31,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:31,544 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into adf06638e87442c18df2ee542892dde5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:31,544 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,544 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=12, startTime=1680807391391; duration=0sec 2023-04-06 18:56:31,544 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:31,545 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:31,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807451544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807451544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807451545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:31,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:31,554 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:31,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:31,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:31,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:31,554 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:31,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,564 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:31,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:31,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:31,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,575 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:31,575 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:31,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f2f88628aade4cf9bbc6e9e6e5d18ceb 2023-04-06 18:56:31,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/95954368e2cc4665863caa51fa1a0376 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376 2023-04-06 18:56:31,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807451591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807451592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376, entries=250, sequenceid=362, filesize=16.7 K 2023-04-06 18:56:31,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/813d2eacd1654240bf536b6d1c6b4e58 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58 2023-04-06 18:56:31,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58, entries=150, sequenceid=362, filesize=11.9 K 2023-04-06 18:56:31,624 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/50292605e68c41c18f37ec0a1cc78941 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/50292605e68c41c18f37ec0a1cc78941 2023-04-06 18:56:31,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f2f88628aade4cf9bbc6e9e6e5d18ceb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb 2023-04-06 18:56:31,636 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 50292605e68c41c18f37ec0a1cc78941(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:31,636 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,636 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=12, startTime=1680807391391; duration=0sec 2023-04-06 18:56:31,636 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:31,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb, entries=150, sequenceid=362, filesize=11.9 K 2023-04-06 18:56:31,648 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 97c1cc7a8c861764bde589e4ad2af6a8 in 256ms, sequenceid=362, compaction requested=false 2023-04-06 18:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,675 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:31,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:31,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:31,682 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:31,682 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,682 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:31,682 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:31,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:31,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:31,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:31,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:31,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:31,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:31,687 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:31,688 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:31,688 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,688 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:31,689 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,733 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/c6c0f6ef6815456ca8836f027d6eba50 2023-04-06 18:56:31,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:31,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:31,804 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/70b2ddb8c1374fdf86b42bba7ccb4926 2023-04-06 18:56:31,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807451799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807451803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807451805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,860 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/7402a864fc83468ab4e01d232ae60db2 2023-04-06 18:56:31,875 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/c6c0f6ef6815456ca8836f027d6eba50 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50 2023-04-06 18:56:31,882 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50, entries=150, sequenceid=379, filesize=11.9 K 2023-04-06 18:56:31,883 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/70b2ddb8c1374fdf86b42bba7ccb4926 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926 2023-04-06 18:56:31,890 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926, entries=150, sequenceid=379, filesize=11.9 K 2023-04-06 18:56:31,891 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/7402a864fc83468ab4e01d232ae60db2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2 2023-04-06 18:56:31,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807451901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807451904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,914 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2, entries=150, sequenceid=379, filesize=11.9 K 2023-04-06 18:56:31,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807451907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,915 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 97c1cc7a8c861764bde589e4ad2af6a8 in 227ms, sequenceid=379, compaction requested=true 2023-04-06 18:56:31,915 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:31,915 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:31,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:31,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:31,920 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:31,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:31,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:31,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:31,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c768437[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:31,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:31,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,927 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:31,927 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:31,927 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:31,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:31,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:31,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:31,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:31,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:31,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:31,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:31,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:31,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:31,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:31,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807451946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:31,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807451948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:31,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:31,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:31,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:31,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:31,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:31,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:31,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:31,953 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:31,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:31,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:31,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d3e21fa2da6a48ed82138f82fa340746 2023-04-06 18:56:32,053 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807452051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807452052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,062 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,063 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,064 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,064 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,068 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d28cda7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,071 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,072 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,075 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,075 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,080 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807452121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,180 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,183 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,184 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,187 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,187 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,190 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@676a39a9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,195 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,195 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,196 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,196 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,197 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,201 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807452255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807452255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,305 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,306 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,307 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,307 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,309 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,309 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6898d1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,313 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,313 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,317 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,317 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,323 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,323 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/baaf16d4eb5e4bad82e32d1124c59b70 2023-04-06 18:56:32,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807452409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807452409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,423 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,426 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52670 deadline: 1680807452425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,427 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,427 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,427 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,431 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f7fe77[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,434 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,435 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,437 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,437 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,437 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,437 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,437 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,439 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,441 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,441 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,542 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,544 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,545 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,545 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,546 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee95a9b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,552 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807452563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807452563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,571 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,571 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,692 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f8939e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,697 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,698 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,703 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,703 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,703 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,804 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/1e33640c825b428098c79c89aff0be56 2023-04-06 18:56:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,807 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,808 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,808 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,808 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,808 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/d3e21fa2da6a48ed82138f82fa340746 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746 2023-04-06 18:56:32,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746, entries=200, sequenceid=399, filesize=14.3 K 2023-04-06 18:56:32,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/baaf16d4eb5e4bad82e32d1124c59b70 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70 2023-04-06 18:56:32,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@617f2440[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70, entries=150, sequenceid=399, filesize=11.9 K 2023-04-06 18:56:32,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/1e33640c825b428098c79c89aff0be56 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56 2023-04-06 18:56:32,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56, entries=150, sequenceid=399, filesize=11.9 K 2023-04-06 18:56:32,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 97c1cc7a8c861764bde589e4ad2af6a8 in 907ms, sequenceid=399, compaction requested=true 2023-04-06 18:56:32,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:32,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,829 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,829 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:32,831 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:32,831 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:32,831 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,831 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/85919e12f70a4b9488572c36e1a8cad4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=48.3 K 2023-04-06 18:56:32,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,832 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:32,832 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 85919e12f70a4b9488572c36e1a8cad4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:32,832 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56814 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,832 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:32,832 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,832 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/adf06638e87442c18df2ee542892dde5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=55.5 K 2023-04-06 18:56:32,833 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 813d2eacd1654240bf536b6d1c6b4e58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680807391254 2023-04-06 18:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,833 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 70b2ddb8c1374fdf86b42bba7ccb4926, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680807391410 2023-04-06 18:56:32,833 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting adf06638e87442c18df2ee542892dde5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,833 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting baaf16d4eb5e4bad82e32d1124c59b70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:32,833 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 95954368e2cc4665863caa51fa1a0376, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680807391254 2023-04-06 18:56:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,834 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c6c0f6ef6815456ca8836f027d6eba50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680807391410 2023-04-06 18:56:32,835 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d3e21fa2da6a48ed82138f82fa340746, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,836 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,837 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,844 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#77 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:32,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,844 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:32,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:32,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,848 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:32,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,848 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:32,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:32,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:32,854 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#78 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:32,890 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/c03f0308bc974cbab49b426ee24f47fe as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c03f0308bc974cbab49b426ee24f47fe 2023-04-06 18:56:32,896 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into c03f0308bc974cbab49b426ee24f47fe(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:32,896 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:32,896 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=12, startTime=1680807392829; duration=0sec 2023-04-06 18:56:32,897 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:32,897 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:32,898 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:32,899 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:32,899 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,899 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/50292605e68c41c18f37ec0a1cc78941, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=48.3 K 2023-04-06 18:56:32,899 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 50292605e68c41c18f37ec0a1cc78941, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680807390624 2023-04-06 18:56:32,899 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f2f88628aade4cf9bbc6e9e6e5d18ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680807391254 2023-04-06 18:56:32,900 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7402a864fc83468ab4e01d232ae60db2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680807391410 2023-04-06 18:56:32,900 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 1e33640c825b428098c79c89aff0be56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:32,907 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:32,918 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f408800bb38345ca82c7fc3a6bdd45c7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f408800bb38345ca82c7fc3a6bdd45c7 2023-04-06 18:56:32,923 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into f408800bb38345ca82c7fc3a6bdd45c7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:32,923 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:32,923 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=12, startTime=1680807392832; duration=0sec 2023-04-06 18:56:32,924 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:32,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:32,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:32,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/18a45a73ae3546c7a0c2dcbbd23bbc3a 2023-04-06 18:56:32,948 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:32,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/6e30e32f4dda4b2993af162086653678 2023-04-06 18:56:32,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:32,966 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:32,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:32,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:32,967 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,967 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,968 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:32,968 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,968 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,968 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:32,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:32,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:32,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:32,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:32,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:32,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:32,983 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:32,983 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:32,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:32,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:32,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:32,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:32,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/d9f16bcdff6444ff8a7dfbf1b0ff5603 2023-04-06 18:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:32,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b616a5e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:32,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:32,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:32,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:32,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:32,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:32,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:32,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:32,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:32,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:32,997 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:32,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:32,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:32,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:32,997 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:32,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:32,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807452995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:32,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/18a45a73ae3546c7a0c2dcbbd23bbc3a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a 2023-04-06 18:56:33,000 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,001 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,001 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,002 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,003 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a, entries=200, sequenceid=414, filesize=14.3 K 2023-04-06 18:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/6e30e32f4dda4b2993af162086653678 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678 2023-04-06 18:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,010 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:33,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678, entries=150, sequenceid=414, filesize=11.9 K 2023-04-06 18:56:33,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/d9f16bcdff6444ff8a7dfbf1b0ff5603 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603 2023-04-06 18:56:33,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603, entries=150, sequenceid=414, filesize=11.9 K 2023-04-06 18:56:33,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 97c1cc7a8c861764bde589e4ad2af6a8 in 91ms, sequenceid=414, compaction requested=false 2023-04-06 18:56:33,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:33,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:33,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:33,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:33,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807453099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807453100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807453101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,114 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,115 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,115 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,115 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,116 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,116 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,116 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,116 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,117 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe8da2b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,119 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,120 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,122 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,122 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,124 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,131 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:33,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,136 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,136 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807453206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807453206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,236 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,240 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,241 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,241 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,242 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,243 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,243 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,243 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,243 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,244 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4300c42b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,248 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,251 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,251 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,257 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,283 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/298c57e52fdc40bb81b4b8d57e22799a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/298c57e52fdc40bb81b4b8d57e22799a 2023-04-06 18:56:33,289 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into 298c57e52fdc40bb81b4b8d57e22799a(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:33,289 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:33,289 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=12, startTime=1680807392829; duration=0sec 2023-04-06 18:56:33,289 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:33,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807453306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,357 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,360 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,363 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,367 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc5b52f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,367 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,368 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,368 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,368 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,369 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,371 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,371 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,371 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,371 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,371 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,373 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:33,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,377 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,377 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807453409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807453409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807453416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807453421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,477 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,480 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,484 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,484 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,484 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,484 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b860793[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,489 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,490 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/ebf08f413def4d18b628bf712d9e71ed 2023-04-06 18:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,493 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,494 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:33,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,510 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,611 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,613 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,613 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,613 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,613 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807453610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,616 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,616 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,616 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,616 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,617 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2236a94a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,620 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,622 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,622 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,623 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,626 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807453712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:33,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807453717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,729 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,738 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,741 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,741 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,741 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,741 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,741 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,742 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,742 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,742 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,742 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,742 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675210bb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,744 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,744 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,745 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,745 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,745 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,746 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,748 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,748 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,752 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,753 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,754 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:33,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:33,857 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:33,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:33,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:33,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:33,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:33,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:33,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:33,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:33,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:33,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:33,863 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:33,863 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:33,863 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:33,863 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:33,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:33,869 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:33,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c8bdb3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:33,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:33,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,875 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,876 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:33,876 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:33,876 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:33,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:33,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:33,881 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:33,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:33,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:33,882 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:33,882 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:33,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:33,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:33,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:33,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:33,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:33,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:33,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:33,892 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:33,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:33,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:33,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:33,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:33,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:33,892 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:33,892 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:33,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/578d03174581431c82b4a22cd358d22c 2023-04-06 18:56:33,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/b9e8708e1b9c4a86830ea269bf6330f1 2023-04-06 18:56:33,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/ebf08f413def4d18b628bf712d9e71ed as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed 2023-04-06 18:56:33,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed, entries=150, sequenceid=435, filesize=11.9 K 2023-04-06 18:56:33,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/578d03174581431c82b4a22cd358d22c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c 2023-04-06 18:56:33,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c, entries=150, sequenceid=435, filesize=11.9 K 2023-04-06 18:56:33,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/b9e8708e1b9c4a86830ea269bf6330f1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1 2023-04-06 18:56:33,992 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:33,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:33,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1, entries=150, sequenceid=435, filesize=11.9 K 2023-04-06 18:56:33,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:33,996 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 97c1cc7a8c861764bde589e4ad2af6a8 in 928ms, sequenceid=435, compaction requested=true 2023-04-06 18:56:33,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:33,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:33,997 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:33,997 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:33,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:33,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:33,999 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:33,999 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:33,999 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:33,999 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:33,999 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:33,999 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:34,000 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c03f0308bc974cbab49b426ee24f47fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=38.9 K 2023-04-06 18:56:34,000 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/298c57e52fdc40bb81b4b8d57e22799a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=36.6 K 2023-04-06 18:56:34,000 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c03f0308bc974cbab49b426ee24f47fe, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:34,000 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 298c57e52fdc40bb81b4b8d57e22799a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:34,000 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 18a45a73ae3546c7a0c2dcbbd23bbc3a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680807391937 2023-04-06 18:56:34,000 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6e30e32f4dda4b2993af162086653678, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680807391937 2023-04-06 18:56:34,001 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ebf08f413def4d18b628bf712d9e71ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:34,001 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 578d03174581431c82b4a22cd358d22c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:34,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:34,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:34,006 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:34,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:34,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:34,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:34,009 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,009 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:34,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,011 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:34,011 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:34,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:34,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:34,012 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:34,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:34,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:34,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:34,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:34,015 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:34,015 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:34,015 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:34,015 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:34,015 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:34,015 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:34,016 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:34,016 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:34,016 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:34,016 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:34,016 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:34,043 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/38b36aea38fd40a98c68d5e9251aa881 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/38b36aea38fd40a98c68d5e9251aa881 2023-04-06 18:56:34,050 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 38b36aea38fd40a98c68d5e9251aa881(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:34,050 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:34,050 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=13, startTime=1680807393997; duration=0sec 2023-04-06 18:56:34,050 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:34,050 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:34,052 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:34,052 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:34,052 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:34,052 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f408800bb38345ca82c7fc3a6bdd45c7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=36.5 K 2023-04-06 18:56:34,053 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f408800bb38345ca82c7fc3a6bdd45c7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1680807391795 2023-04-06 18:56:34,053 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d9f16bcdff6444ff8a7dfbf1b0ff5603, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680807391937 2023-04-06 18:56:34,053 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b9e8708e1b9c4a86830ea269bf6330f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:34,060 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/b29af426a6264ca99c694737fe5b5ab2 2023-04-06 18:56:34,069 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#89 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:34,080 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/12e3b7c72dc14da299600924a43323e9 2023-04-06 18:56:34,085 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/0977c86e088f47dd8d58e4bb5a3bae8d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0977c86e088f47dd8d58e4bb5a3bae8d 2023-04-06 18:56:34,097 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 0977c86e088f47dd8d58e4bb5a3bae8d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:34,097 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:34,097 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=13, startTime=1680807393997; duration=0sec 2023-04-06 18:56:34,097 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:34,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:34,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807394141, max timestamp=1680807394163Num uniques 859; 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807394141, max timestamp=1680807394163Num uniques 859; 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807394141, max timestamp=1680807394163Num uniques 859; 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:34,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:34,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807454221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807454222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807454223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807454327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807454328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,433 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/4a00062c557b425ca22927d1f2af80a4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/4a00062c557b425ca22927d1f2af80a4 2023-04-06 18:56:34,438 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into 4a00062c557b425ca22927d1f2af80a4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:34,438 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:34,438 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=13, startTime=1680807393997; duration=0sec 2023-04-06 18:56:34,439 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:34,501 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/390dd5b3d87948a2a0a5e831101ac3d7 2023-04-06 18:56:34,514 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/b29af426a6264ca99c694737fe5b5ab2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2 2023-04-06 18:56:34,519 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2, entries=150, sequenceid=447, filesize=11.9 K 2023-04-06 18:56:34,520 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/12e3b7c72dc14da299600924a43323e9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9 2023-04-06 18:56:34,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807454532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807454534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,542 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9, entries=150, sequenceid=447, filesize=11.9 K 2023-04-06 18:56:34,544 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/390dd5b3d87948a2a0a5e831101ac3d7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7 2023-04-06 18:56:34,552 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7, entries=150, sequenceid=447, filesize=11.9 K 2023-04-06 18:56:34,553 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 97c1cc7a8c861764bde589e4ad2af6a8 in 538ms, sequenceid=447, compaction requested=false 2023-04-06 18:56:34,554 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:34,554 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:34,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,556 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:34,556 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,556 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:34,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:34,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae98a86[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:34,558 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:34,558 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,559 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,559 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:34,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:34,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:34,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:34,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:34,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:34,565 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:34,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:34,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:34,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:34,565 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:34,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:34,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:34,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:34,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:34,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:34,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:34,581 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:34,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:34,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:34,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:34,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:34,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:34,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:34,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:34,685 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:34,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:34,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:34,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:34,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:34,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:34,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:34,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:34,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:34,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:34,691 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-06 18:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:34,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:34,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:34,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:34,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:34,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:34,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:34,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807454852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807454855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:34,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807454957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:34,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807454958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,131 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a38ede4367a94cfea2b8fa146d96a018 2023-04-06 18:56:35,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807455161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807455161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807455229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52636 deadline: 1680807455432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,437 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:35,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52664 deadline: 1680807455438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,443 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:35,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807455467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:35,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807455473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,584 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/1eb97469897e42c5ae4c06f413ce774a 2023-04-06 18:56:35,643 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3 2023-04-06 18:56:35,652 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a38ede4367a94cfea2b8fa146d96a018 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018 2023-04-06 18:56:35,659 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018, entries=150, sequenceid=485, filesize=11.9 K 2023-04-06 18:56:35,661 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/1eb97469897e42c5ae4c06f413ce774a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a 2023-04-06 18:56:35,673 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a, entries=150, sequenceid=485, filesize=11.9 K 2023-04-06 18:56:35,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3 2023-04-06 18:56:35,679 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3, entries=150, sequenceid=485, filesize=11.9 K 2023-04-06 18:56:35,680 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 97c1cc7a8c861764bde589e4ad2af6a8 in 989ms, sequenceid=485, compaction requested=true 2023-04-06 18:56:35,680 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:35,680 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:35,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:35,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:35,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:35,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:35,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:35,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d020e68[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:35,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:35,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:35,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:35,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:35,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:35,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:35,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:35,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:35,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:35,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:35,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:35,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:35,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:35,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:35,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:35,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:35,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:35,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:35,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:35,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,794 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:35,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:35,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:35,805 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:35,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:35,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:35,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:35,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,807 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:35,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:35,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:35,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:35,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:35,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:35,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:35,810 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:35,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:35,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:35,811 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:35,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:35,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:35,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:35,836 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/05a509bd44a24b8290a8f902e3e76803 2023-04-06 18:56:35,856 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/115cea25a6a44db48f324ddaee8edc0d 2023-04-06 18:56:35,872 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/e3151b2b724744bc9a0436cea3dbf387 2023-04-06 18:56:35,878 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/05a509bd44a24b8290a8f902e3e76803 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803 2023-04-06 18:56:35,883 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803, entries=150, sequenceid=496, filesize=11.9 K 2023-04-06 18:56:35,884 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/115cea25a6a44db48f324ddaee8edc0d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d 2023-04-06 18:56:35,889 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d, entries=150, sequenceid=496, filesize=11.9 K 2023-04-06 18:56:35,890 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/e3151b2b724744bc9a0436cea3dbf387 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387 2023-04-06 18:56:35,894 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387, entries=150, sequenceid=496, filesize=11.9 K 2023-04-06 18:56:35,895 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 97c1cc7a8c861764bde589e4ad2af6a8 in 85ms, sequenceid=496, compaction requested=true 2023-04-06 18:56:35,895 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:35,895 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:35,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:35,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:35,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,895 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:35,896 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:35,897 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,897 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,898 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,898 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:35,898 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2202bb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:35,899 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,899 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:35,899 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:35,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:35,900 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,900 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:35,900 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:35,901 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:35,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:35,903 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:35,903 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:35,903 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:35,903 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:35,904 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:35,905 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,905 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:35,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,905 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:35,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:35,906 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:35,907 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:35,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:35,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:35,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:35,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:35,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:35,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:35,910 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:35,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:35,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:35,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:35,989 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:36,011 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,019 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,019 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,019 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,019 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,020 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,020 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,021 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,022 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,022 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,026 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,036 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,036 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,036 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,036 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,037 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed30f69[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,042 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,043 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,043 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,044 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,044 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,044 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807396030, max timestamp=1680807396044Num uniques 859; 2023-04-06 18:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,047 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,047 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807396030, max timestamp=1680807396044Num uniques 859; 2023-04-06 18:56:36,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:36,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807396030, max timestamp=1680807396044Num uniques 859; 2023-04-06 18:56:36,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:36,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,054 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,062 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,062 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,062 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807456076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807456082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,164 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,168 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807456186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807456187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,197 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,198 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,198 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,198 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78281f8f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,239 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,240 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,240 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,243 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,243 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,243 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,243 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,243 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,368 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,369 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,376 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,376 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,376 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,376 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807456391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d43fac[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,397 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,397 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,399 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807456391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/375eae1036f24e78947050bb3497fc44 2023-04-06 18:56:36,418 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,418 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,419 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,420 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:36,520 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,524 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,525 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f99dc77[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,538 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,538 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,543 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,546 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,548 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,548 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,551 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:36,554 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,554 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,654 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,658 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,659 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,659 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,659 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,675 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,675 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,675 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291732aa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,683 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,683 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,690 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807456692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807456706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,790 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,792 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,794 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,795 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,795 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,795 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,795 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,795 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117b8460[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,798 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,799 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,799 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,799 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,799 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,799 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,799 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,801 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,801 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,801 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,801 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,802 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,802 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,802 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,802 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,805 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,805 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:36,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:36,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/2c787b4b547b4354ac944f6634a0faa0 2023-04-06 18:56:36,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:36,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:36,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:36,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:36,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:36,910 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:36,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:36,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:36,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,911 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:36,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:36,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:36,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:36,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:36,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:36,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:36,913 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:36,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4586e0bc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:36,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:36,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,915 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:36,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:36,915 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,915 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:36,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:36,916 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:36,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:36,918 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:36,918 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:36,918 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:36,918 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:36,918 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:36,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:36,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:36,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:36,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:36,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:36,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:36,927 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:36,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:36,927 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:37,027 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:37,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:37,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:37,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:37,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:37,030 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:37,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:37,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:37,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,031 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:37,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:37,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:37,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:37,033 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,033 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:37,033 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:37,033 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:37,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:37,034 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43df61f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:37,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:37,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,036 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:37,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:37,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:37,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:37,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:37,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:37,040 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:37,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:37,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:37,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:37,040 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:37,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,041 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:37,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,046 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:37,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:37,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:37,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:37,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:37,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:37,150 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:37,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:37,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:37,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:37,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:37,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:37,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:37,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:37,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:37,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:37,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:37,164 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:37,164 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:37,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:37,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:37,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:37,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:37,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:37,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:37,168 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,168 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:37,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142f0650[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:37,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:37,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,176 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:37,176 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:37,176 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:37,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:37,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,177 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:37,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:37,180 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:37,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:37,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:37,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:37,180 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:37,181 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,181 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,182 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:37,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:37,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:37,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,195 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,195 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:37,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:37,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807457197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:37,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807457211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:37,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52672 deadline: 1680807457237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,240 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:37,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/7104242603ce4ca3a9885a16cd356f43 2023-04-06 18:56:37,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/375eae1036f24e78947050bb3497fc44 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44 2023-04-06 18:56:37,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44, entries=250, sequenceid=507, filesize=16.7 K 2023-04-06 18:56:37,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/2c787b4b547b4354ac944f6634a0faa0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0 2023-04-06 18:56:37,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0, entries=150, sequenceid=507, filesize=11.9 K 2023-04-06 18:56:37,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/7104242603ce4ca3a9885a16cd356f43 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43 2023-04-06 18:56:37,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:37,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:37,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:37,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43, entries=150, sequenceid=507, filesize=11.9 K 2023-04-06 18:56:37,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:37,308 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:37,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:37,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:37,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 97c1cc7a8c861764bde589e4ad2af6a8 in 1320ms, sequenceid=507, compaction requested=true 2023-04-06 18:56:37,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:37,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:37,308 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:37,308 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:37,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:37,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,308 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:37,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,314 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:37,316 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:37,316 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:37,316 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,316 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/4a00062c557b425ca22927d1f2af80a4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=60.5 K 2023-04-06 18:56:37,316 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66798 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:37,316 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:37,316 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,317 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/38b36aea38fd40a98c68d5e9251aa881, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=65.2 K 2023-04-06 18:56:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:37,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:37,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:37,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:37,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:37,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:37,317 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4a00062c557b425ca22927d1f2af80a4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:37,318 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 12e3b7c72dc14da299600924a43323e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680807393092 2023-04-06 18:56:37,318 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 38b36aea38fd40a98c68d5e9251aa881, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:37,318 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1eb97469897e42c5ae4c06f413ce774a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680807394213 2023-04-06 18:56:37,318 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b29af426a6264ca99c694737fe5b5ab2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680807393092 2023-04-06 18:56:37,320 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 115cea25a6a44db48f324ddaee8edc0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680807394849 2023-04-06 18:56:37,321 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a38ede4367a94cfea2b8fa146d96a018, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680807394213 2023-04-06 18:56:37,321 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2c787b4b547b4354ac944f6634a0faa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395985 2023-04-06 18:56:37,322 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 05a509bd44a24b8290a8f902e3e76803, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680807394849 2023-04-06 18:56:37,323 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 375eae1036f24e78947050bb3497fc44, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395974 2023-04-06 18:56:37,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:37,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:37,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:37,334 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-06 18:56:37,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:37,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:37,336 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:37,336 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:37,336 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:37,336 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:37,337 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#101 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:37,352 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#102 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:37,379 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/b70ce8942af4422fbfd208a26b280d35 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/b70ce8942af4422fbfd208a26b280d35 2023-04-06 18:56:37,386 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into b70ce8942af4422fbfd208a26b280d35(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:37,387 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:37,387 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=11, startTime=1680807397308; duration=0sec 2023-04-06 18:56:37,387 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:37,387 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:56:37,395 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:56:37,395 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:37,396 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,396 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0977c86e088f47dd8d58e4bb5a3bae8d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=60.4 K 2023-04-06 18:56:37,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0977c86e088f47dd8d58e4bb5a3bae8d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680807392974 2023-04-06 18:56:37,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 390dd5b3d87948a2a0a5e831101ac3d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680807393092 2023-04-06 18:56:37,397 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9fb4e84fc8cb4e8eaf0c1697480d6eb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680807394213 2023-04-06 18:56:37,397 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e3151b2b724744bc9a0436cea3dbf387, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680807394849 2023-04-06 18:56:37,397 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7104242603ce4ca3a9885a16cd356f43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395985 2023-04-06 18:56:37,414 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/3fdbaacf026c46368da4a3d99865304b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3fdbaacf026c46368da4a3d99865304b 2023-04-06 18:56:37,420 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#104 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:37,429 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/4a3db7faf9c5468e83df87c6e122fa84 2023-04-06 18:56:37,431 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into 3fdbaacf026c46368da4a3d99865304b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:37,431 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:37,431 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=11, startTime=1680807397308; duration=0sec 2023-04-06 18:56:37,431 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:37,469 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f05903d8199a4cb4b9c60458ba7e05fc 2023-04-06 18:56:37,865 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/658bfeca81ca45619558076ac9a43606 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/658bfeca81ca45619558076ac9a43606 2023-04-06 18:56:37,870 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 658bfeca81ca45619558076ac9a43606(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:37,870 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:37,870 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=11, startTime=1680807397308; duration=0sec 2023-04-06 18:56:37,870 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:37,886 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/2cacb13219cc44be99376eceecf97464 2023-04-06 18:56:37,895 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/4a3db7faf9c5468e83df87c6e122fa84 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84 2023-04-06 18:56:37,902 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84, entries=150, sequenceid=541, filesize=11.9 K 2023-04-06 18:56:37,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f05903d8199a4cb4b9c60458ba7e05fc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc 2023-04-06 18:56:37,913 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc, entries=150, sequenceid=541, filesize=11.9 K 2023-04-06 18:56:37,916 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/2cacb13219cc44be99376eceecf97464 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464 2023-04-06 18:56:37,922 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464, entries=150, sequenceid=541, filesize=11.9 K 2023-04-06 18:56:37,923 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 97c1cc7a8c861764bde589e4ad2af6a8 in 589ms, sequenceid=541, compaction requested=false 2023-04-06 18:56:37,923 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:37,924 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:37,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:37,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593ad831[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:37,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:37,931 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:37,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:37,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:37,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:37,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:37,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:37,934 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:37,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:37,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:37,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:37,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:37,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:37,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:37,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:37,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:37,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:37,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:37,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:37,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:37,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:37,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:37,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:37,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:37,952 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:37,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:37,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:37,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,052 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,055 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,056 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:38,067 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:38,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:38,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bcb50aa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:38,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:38,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,072 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:38,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:38,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:38,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:38,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:38,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:38,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:38,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:38,078 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:38,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:38,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:38,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:38,078 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:38,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:38,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,097 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:38,097 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:38,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,197 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,210 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,210 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,210 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,210 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:38,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:38,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:38,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:38,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:38,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:38,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,238 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,238 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,238 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:38,238 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:38,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:38,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:38,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:38,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:38,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402dd6c2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:38,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:38,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:38,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:38,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:38,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:38,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:38,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807398235, max timestamp=1680807398263Num uniques 859; 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807398235, max timestamp=1680807398263Num uniques 859; 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807398235, max timestamp=1680807398263Num uniques 859; 2023-04-06 18:56:38,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:38,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:38,268 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:38,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:38,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:38,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:38,268 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:38,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/185c03108a3d480ba608ca2e64e25d40 2023-04-06 18:56:38,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,293 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:38,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:38,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:38,293 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:38,293 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:38,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/d3dd16e53c984b99988b8198fe08c521 2023-04-06 18:56:38,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807458335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807458337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,393 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,397 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,399 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,400 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,400 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,400 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:38,400 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:38,400 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:38,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5c4511[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:38,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:38,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,404 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:38,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:38,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:38,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:38,405 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:38,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:38,410 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:38,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:38,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:38,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:38,411 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:38,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,418 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:38,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:38,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:38,428 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:38,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:38,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807458440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807458442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,529 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,533 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,534 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:38,539 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:38,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:38,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:38,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:38,540 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,540 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,540 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,540 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:38,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2328ebfb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:38,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:38,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,545 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:38,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:38,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:38,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:38,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:38,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:38,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:38,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:38,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:38,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:38,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:38,556 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:38,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:38,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:38,569 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:38,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,570 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:38,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52670 deadline: 1680807458642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:38,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52638 deadline: 1680807458645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,673 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,675 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,675 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,675 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,677 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,677 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,677 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:38,677 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:38,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,678 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:38,678 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:38,679 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:38,682 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5217b134[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:38,682 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:38,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:38,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:38,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:38,686 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:38,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:38,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:38,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:38,687 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:38,688 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,688 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,688 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:38,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:38,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:38,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:38,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:38,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:38,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:38,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:38,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:38,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/611ba3f7b0f54361afeceff1144f750e 2023-04-06 18:56:38,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/185c03108a3d480ba608ca2e64e25d40 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40 2023-04-06 18:56:38,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40, entries=150, sequenceid=555, filesize=11.9 K 2023-04-06 18:56:38,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/d3dd16e53c984b99988b8198fe08c521 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521 2023-04-06 18:56:38,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521, entries=150, sequenceid=555, filesize=11.9 K 2023-04-06 18:56:38,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/611ba3f7b0f54361afeceff1144f750e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e 2023-04-06 18:56:38,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e, entries=150, sequenceid=555, filesize=11.9 K 2023-04-06 18:56:38,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 97c1cc7a8c861764bde589e4ad2af6a8 in 561ms, sequenceid=555, compaction requested=true 2023-04-06 18:56:38,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:38,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:38,780 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:38,780 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:38,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:38,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:38,781 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:38,781 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/A is initiating minor compaction (all files) 2023-04-06 18:56:38,781 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:38,781 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/A in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,781 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/B is initiating minor compaction (all files) 2023-04-06 18:56:38,781 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/B in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,781 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3fdbaacf026c46368da4a3d99865304b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=36.8 K 2023-04-06 18:56:38,781 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/b70ce8942af4422fbfd208a26b280d35, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=36.8 K 2023-04-06 18:56:38,782 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 3fdbaacf026c46368da4a3d99865304b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395985 2023-04-06 18:56:38,782 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b70ce8942af4422fbfd208a26b280d35, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395985 2023-04-06 18:56:38,782 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4a3db7faf9c5468e83df87c6e122fa84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680807396066 2023-04-06 18:56:38,782 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f05903d8199a4cb4b9c60458ba7e05fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680807396066 2023-04-06 18:56:38,783 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 185c03108a3d480ba608ca2e64e25d40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680807398212 2023-04-06 18:56:38,783 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d3dd16e53c984b99988b8198fe08c521, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680807398212 2023-04-06 18:56:38,789 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#A#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:38,791 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#B#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:38,794 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:38,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:38,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:38,798 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:38,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:38,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:38,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:38,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:38,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:38,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:38,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:38,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:38,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:38,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:38,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:38,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:38,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:38,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:38,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. started... 2023-04-06 18:56:38,812 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 18:56:38,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:38,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:38,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:38,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:38,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:38,813 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:38,821 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/dab47be42e4346f6bf1c1b6531ef71a3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/dab47be42e4346f6bf1c1b6531ef71a3 2023-04-06 18:56:38,826 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/B of 97c1cc7a8c861764bde589e4ad2af6a8 into dab47be42e4346f6bf1c1b6531ef71a3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:38,826 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:38,826 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/B, priority=13, startTime=1680807398780; duration=0sec 2023-04-06 18:56:38,827 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:38,827 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:38,830 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:38,830 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 97c1cc7a8c861764bde589e4ad2af6a8/C is initiating minor compaction (all files) 2023-04-06 18:56:38,830 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 97c1cc7a8c861764bde589e4ad2af6a8/C in TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:38,831 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/658bfeca81ca45619558076ac9a43606, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp, totalSize=36.8 K 2023-04-06 18:56:38,832 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 658bfeca81ca45619558076ac9a43606, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1680807395985 2023-04-06 18:56:38,832 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2cacb13219cc44be99376eceecf97464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680807396066 2023-04-06 18:56:38,832 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 611ba3f7b0f54361afeceff1144f750e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1680807398212 2023-04-06 18:56:38,840 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 97c1cc7a8c861764bde589e4ad2af6a8#C#compaction#113 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:38,878 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e044e22 to 127.0.0.1:63695 2023-04-06 18:56:38,878 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:38,879 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18f7ec13 to 127.0.0.1:63695 2023-04-06 18:56:38,879 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:38,884 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:63695 2023-04-06 18:56:38,884 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:38,885 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ab9467e to 127.0.0.1:63695 2023-04-06 18:56:38,885 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:38,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:38,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. as already flushing 2023-04-06 18:56:38,947 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ecabb06 to 127.0.0.1:63695 2023-04-06 18:56:38,947 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:38,953 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ca212c to 127.0.0.1:63695 2023-04-06 18:56:38,954 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:39,208 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/bc5fb43ee21b488d8ef0cf7205a5aaff as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/bc5fb43ee21b488d8ef0cf7205a5aaff 2023-04-06 18:56:39,218 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/A of 97c1cc7a8c861764bde589e4ad2af6a8 into bc5fb43ee21b488d8ef0cf7205a5aaff(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:39,218 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:39,219 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/A, priority=13, startTime=1680807398780; duration=0sec 2023-04-06 18:56:39,219 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:39,233 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/3853461f74f6446c9cf6506372bef99d 2023-04-06 18:56:39,266 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/9b6452e3155e4dbf90a9ef9e81ea23e3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9b6452e3155e4dbf90a9ef9e81ea23e3 2023-04-06 18:56:39,271 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 97c1cc7a8c861764bde589e4ad2af6a8/C of 97c1cc7a8c861764bde589e4ad2af6a8 into 9b6452e3155e4dbf90a9ef9e81ea23e3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:39,271 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:39,271 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8., storeName=97c1cc7a8c861764bde589e4ad2af6a8/C, priority=13, startTime=1680807398780; duration=0sec 2023-04-06 18:56:39,272 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:39,440 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:63695 2023-04-06 18:56:39,441 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:39,468 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:63695 2023-04-06 18:56:39,468 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:39,651 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f3cef5955d0c425dbb1c482711cbd73d 2023-04-06 18:56:40,067 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f95cc11fd3534d62af6eba3e5c9c97e1 2023-04-06 18:56:40,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/3853461f74f6446c9cf6506372bef99d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3853461f74f6446c9cf6506372bef99d 2023-04-06 18:56:40,079 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3853461f74f6446c9cf6506372bef99d, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 18:56:40,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/f3cef5955d0c425dbb1c482711cbd73d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3cef5955d0c425dbb1c482711cbd73d 2023-04-06 18:56:40,088 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3cef5955d0c425dbb1c482711cbd73d, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 18:56:40,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/f95cc11fd3534d62af6eba3e5c9c97e1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f95cc11fd3534d62af6eba3e5c9c97e1 2023-04-06 18:56:40,096 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f95cc11fd3534d62af6eba3e5c9c97e1, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 18:56:40,096 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=26.84 KB/27480 for 97c1cc7a8c861764bde589e4ad2af6a8 in 1284ms, sequenceid=591, compaction requested=false 2023-04-06 18:56:40,096 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:40,096 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:40,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:40,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:40,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,098 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:40,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:40,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca2f672[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:40,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:40,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:40,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:40,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:40,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:40,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:40,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:40,104 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:40,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:40,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:40,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:40,104 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:40,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:40,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:40,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:40,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:40,109 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:40,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:40,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:40,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:40,110 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:40,110 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:40,210 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:40,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:41,255 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45864842 to 127.0.0.1:63695 2023-04-06 18:56:41,255 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:41,255 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 152 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4395 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4303 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1903 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5709 rows 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1916 2023-04-06 18:56:41,256 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5745 rows 2023-04-06 18:56:41,257 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:56:41,257 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x490ff358 to 127.0.0.1:63695 2023-04-06 18:56:41,257 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:56:41,262 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:56:41,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:56:41,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:41,275 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807401275"}]},"ts":"1680807401275"} 2023-04-06 18:56:41,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:41,276 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:56:41,278 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:56:41,279 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, UNASSIGN}] 2023-04-06 18:56:41,281 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, UNASSIGN 2023-04-06 18:56:41,282 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=97c1cc7a8c861764bde589e4ad2af6a8, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:41,282 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807401282"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807401282"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807401282"}]},"ts":"1680807401282"} 2023-04-06 18:56:41,284 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:41,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:41,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:41,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 97c1cc7a8c861764bde589e4ad2af6a8, disabling compactions & flushes 2023-04-06 18:56:41,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:41,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:41,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. after waiting 0 ms 2023-04-06 18:56:41,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:41,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 97c1cc7a8c861764bde589e4ad2af6a8 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 18:56:41,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=A 2023-04-06 18:56:41,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:41,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=B 2023-04-06 18:56:41,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:41,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 97c1cc7a8c861764bde589e4ad2af6a8, store=C 2023-04-06 18:56:41,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:41,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a49533e84fd941a096bf774261ef4ccf 2023-04-06 18:56:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:42,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/fd48cea0044640baa8d9d0b3a236c48c 2023-04-06 18:56:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:42,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/e07feed4c2d646e68d5acb4280bbc377 2023-04-06 18:56:42,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/A/a49533e84fd941a096bf774261ef4ccf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a49533e84fd941a096bf774261ef4ccf 2023-04-06 18:56:42,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a49533e84fd941a096bf774261ef4ccf, entries=150, sequenceid=602, filesize=11.9 K 2023-04-06 18:56:42,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/B/fd48cea0044640baa8d9d0b3a236c48c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fd48cea0044640baa8d9d0b3a236c48c 2023-04-06 18:56:42,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fd48cea0044640baa8d9d0b3a236c48c, entries=150, sequenceid=602, filesize=11.9 K 2023-04-06 18:56:42,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/.tmp/C/e07feed4c2d646e68d5acb4280bbc377 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e07feed4c2d646e68d5acb4280bbc377 2023-04-06 18:56:42,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e07feed4c2d646e68d5acb4280bbc377, entries=150, sequenceid=602, filesize=11.9 K 2023-04-06 18:56:42,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 97c1cc7a8c861764bde589e4ad2af6a8 in 1278ms, sequenceid=602, compaction requested=true 2023-04-06 18:56:42,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/7b050d285bf3427d9fb2652097dcbcd2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4bf005a4bbc24e84b31e6e34bab1119f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/98eb426534fc4996bb015e781f25cbdc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/03558ef0f1ea4b88a13a3270085eab32, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/beb810620a0443aebdf799d222910f96, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/adf06638e87442c18df2ee542892dde5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c03f0308bc974cbab49b426ee24f47fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/38b36aea38fd40a98c68d5e9251aa881, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3fdbaacf026c46368da4a3d99865304b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40] to archive 2023-04-06 18:56:42,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:56:42,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f226f2dae102401783a5540eafbc8102 2023-04-06 18:56:42,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f576d36cc5da440aa5650f31ae158837 2023-04-06 18:56:42,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/78f7d7c43ef54bdf86e3528c2504ae7f 2023-04-06 18:56:42,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/7b050d285bf3427d9fb2652097dcbcd2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/7b050d285bf3427d9fb2652097dcbcd2 2023-04-06 18:56:42,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/077fc4a03ac44313b1649934eaaac67c 2023-04-06 18:56:42,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/92c19baa35f04c9e9ceb1b35169bb385 2023-04-06 18:56:42,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/0dcb22003ee84b839b36b2669defcab5 2023-04-06 18:56:42,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4bf005a4bbc24e84b31e6e34bab1119f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4bf005a4bbc24e84b31e6e34bab1119f 2023-04-06 18:56:42,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a2c10beb6d6a49f7b14feff2439dba6a 2023-04-06 18:56:42,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/f305cba27b56488f8f3a0a486d21e521 2023-04-06 18:56:42,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d754bf7989ac41c1954d590cdf911b94 2023-04-06 18:56:42,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/98eb426534fc4996bb015e781f25cbdc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/98eb426534fc4996bb015e781f25cbdc 2023-04-06 18:56:42,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/2517ac63cb634d7ca74c192a392e1400 2023-04-06 18:56:42,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/66cd44b3d6984a54bed3086fa7cc93ca 2023-04-06 18:56:42,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/03558ef0f1ea4b88a13a3270085eab32 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/03558ef0f1ea4b88a13a3270085eab32 2023-04-06 18:56:42,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a42c30e798de46a6ae8c6b2181541cb4 2023-04-06 18:56:42,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/58be823d6ec84af895933737a92f3ea8 2023-04-06 18:56:42,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d05e5b3ed44c4e84ba212e23107e4f96 2023-04-06 18:56:42,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/beb810620a0443aebdf799d222910f96 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/beb810620a0443aebdf799d222910f96 2023-04-06 18:56:42,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/22a8b6f242f94b70bc75b05523a834cd 2023-04-06 18:56:42,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ec749d2eace642c58cc64ba46b3f7032 2023-04-06 18:56:42,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/caa971277c5a4067baeb0891fe34d409 2023-04-06 18:56:42,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/adf06638e87442c18df2ee542892dde5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/adf06638e87442c18df2ee542892dde5 2023-04-06 18:56:42,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/95954368e2cc4665863caa51fa1a0376 2023-04-06 18:56:42,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c6c0f6ef6815456ca8836f027d6eba50 2023-04-06 18:56:42,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/d3e21fa2da6a48ed82138f82fa340746 2023-04-06 18:56:42,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c03f0308bc974cbab49b426ee24f47fe to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/c03f0308bc974cbab49b426ee24f47fe 2023-04-06 18:56:42,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/18a45a73ae3546c7a0c2dcbbd23bbc3a 2023-04-06 18:56:42,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/38b36aea38fd40a98c68d5e9251aa881 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/38b36aea38fd40a98c68d5e9251aa881 2023-04-06 18:56:42,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/ebf08f413def4d18b628bf712d9e71ed 2023-04-06 18:56:42,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/b29af426a6264ca99c694737fe5b5ab2 2023-04-06 18:56:42,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a38ede4367a94cfea2b8fa146d96a018 2023-04-06 18:56:42,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/05a509bd44a24b8290a8f902e3e76803 2023-04-06 18:56:42,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/375eae1036f24e78947050bb3497fc44 2023-04-06 18:56:42,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3fdbaacf026c46368da4a3d99865304b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3fdbaacf026c46368da4a3d99865304b 2023-04-06 18:56:42,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/4a3db7faf9c5468e83df87c6e122fa84 2023-04-06 18:56:42,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/185c03108a3d480ba608ca2e64e25d40 2023-04-06 18:56:42,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c3817c423a2a48a7ad927187529a0d3a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e3ba2056bcd4481a840e9837b97e970a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d37595c1789e4bfe88e0911de27bc2fb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fc480e6c14074b988173e29611517cc1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/be31f6145b184ebe8ee658e796784af1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/85919e12f70a4b9488572c36e1a8cad4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/298c57e52fdc40bb81b4b8d57e22799a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/4a00062c557b425ca22927d1f2af80a4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/b70ce8942af4422fbfd208a26b280d35, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521] to archive 2023-04-06 18:56:42,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:56:42,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/cfdcb49a86254443a3947ffc609f237e 2023-04-06 18:56:42,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/7d897bafb38647ad9b8a1f0c6642a115 2023-04-06 18:56:42,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c3817c423a2a48a7ad927187529a0d3a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c3817c423a2a48a7ad927187529a0d3a 2023-04-06 18:56:42,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1e23c30f4d8e485c9d54a4c9e2d2de7d 2023-04-06 18:56:42,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/da7c8f20e43e48babe1665e4f879489c 2023-04-06 18:56:42,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/9ab829cb0c3a4fa387d48f861b5b043b 2023-04-06 18:56:42,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/c66e76b824c049e0b9331ace7cef1775 2023-04-06 18:56:42,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e3ba2056bcd4481a840e9837b97e970a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e3ba2056bcd4481a840e9837b97e970a 2023-04-06 18:56:42,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/e16449e9cc2e4b51a3aa0aa19d0134a7 2023-04-06 18:56:42,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/a5be464d748043ca9804cd7062c06de1 2023-04-06 18:56:42,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d37595c1789e4bfe88e0911de27bc2fb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d37595c1789e4bfe88e0911de27bc2fb 2023-04-06 18:56:42,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/eab1a8ccf9ed4ccd8153af479e4600e6 2023-04-06 18:56:42,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/212540bffc944ceea81c447af3ca52f2 2023-04-06 18:56:42,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fc480e6c14074b988173e29611517cc1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fc480e6c14074b988173e29611517cc1 2023-04-06 18:56:42,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/31cefa718bb54df7b14168adf30adf5f 2023-04-06 18:56:42,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/50b56f8a85c649f98318fe4ca00fdecb 2023-04-06 18:56:42,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3d0472bedcb4507acbb1bce8b6ebcba 2023-04-06 18:56:42,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/be31f6145b184ebe8ee658e796784af1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/be31f6145b184ebe8ee658e796784af1 2023-04-06 18:56:42,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/53e1b1dc2f78455b8eebdcb5548016a0 2023-04-06 18:56:42,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/16610ce8fa6b4f47b41ab1dc7d8bc919 2023-04-06 18:56:42,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/caf414529cc140ca8f26726582e45dcb 2023-04-06 18:56:42,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/85919e12f70a4b9488572c36e1a8cad4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/85919e12f70a4b9488572c36e1a8cad4 2023-04-06 18:56:42,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/165dee436b9346b48b8847c9767e5a06 2023-04-06 18:56:42,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/813d2eacd1654240bf536b6d1c6b4e58 2023-04-06 18:56:42,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/70b2ddb8c1374fdf86b42bba7ccb4926 2023-04-06 18:56:42,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/298c57e52fdc40bb81b4b8d57e22799a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/298c57e52fdc40bb81b4b8d57e22799a 2023-04-06 18:56:42,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/baaf16d4eb5e4bad82e32d1124c59b70 2023-04-06 18:56:42,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/6e30e32f4dda4b2993af162086653678 2023-04-06 18:56:42,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/4a00062c557b425ca22927d1f2af80a4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/4a00062c557b425ca22927d1f2af80a4 2023-04-06 18:56:42,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/578d03174581431c82b4a22cd358d22c 2023-04-06 18:56:42,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/12e3b7c72dc14da299600924a43323e9 2023-04-06 18:56:42,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/1eb97469897e42c5ae4c06f413ce774a 2023-04-06 18:56:42,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/115cea25a6a44db48f324ddaee8edc0d 2023-04-06 18:56:42,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/b70ce8942af4422fbfd208a26b280d35 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/b70ce8942af4422fbfd208a26b280d35 2023-04-06 18:56:42,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/2c787b4b547b4354ac944f6634a0faa0 2023-04-06 18:56:42,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f05903d8199a4cb4b9c60458ba7e05fc 2023-04-06 18:56:42,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/d3dd16e53c984b99988b8198fe08c521 2023-04-06 18:56:42,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7dd955a6cd8b43639d4479db5c8138da, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/c090e1d7be66470db8eb8161ce2cd5f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0f3558f56411422ebc52e6f37f44cedd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/bd24f13b199d41dfbbbc37e95dcf9607, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/50292605e68c41c18f37ec0a1cc78941, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f408800bb38345ca82c7fc3a6bdd45c7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0977c86e088f47dd8d58e4bb5a3bae8d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/658bfeca81ca45619558076ac9a43606, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e] to archive 2023-04-06 18:56:42,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:56:42,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/46290291ddfc4c1589fecc3a6db07e77 2023-04-06 18:56:42,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/86c8f052d5ab4e659d3665cc3bc8a893 2023-04-06 18:56:42,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7dd955a6cd8b43639d4479db5c8138da to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7dd955a6cd8b43639d4479db5c8138da 2023-04-06 18:56:42,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a841390e4cdb4ef4be7303363084ad6a 2023-04-06 18:56:42,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f34d57524b6647eabbaa57db59bba309 2023-04-06 18:56:42,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/afa928f0db3a449691fff30847e7ea63 2023-04-06 18:56:42,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9c0f2380ad98429e9e567b82de9b775e 2023-04-06 18:56:42,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/c090e1d7be66470db8eb8161ce2cd5f6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/c090e1d7be66470db8eb8161ce2cd5f6 2023-04-06 18:56:42,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1d6528c5826c487eac5401af7a86c8c8 2023-04-06 18:56:42,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/66b50a3783654107b30f8de78109769f 2023-04-06 18:56:42,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/8d6a92c91ca24c72b0f836319a2c70ef 2023-04-06 18:56:42,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0f3558f56411422ebc52e6f37f44cedd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0f3558f56411422ebc52e6f37f44cedd 2023-04-06 18:56:42,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/304e67780c1a400bbb1b7af36e8413a9 2023-04-06 18:56:42,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/80e59de30d6b4396a15dacb23034b016 2023-04-06 18:56:42,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/661847caca294ee5a0691fee30de1de1 2023-04-06 18:56:42,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/3236c9cc7a6e4353b03f792e7349b1cc 2023-04-06 18:56:42,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/bd24f13b199d41dfbbbc37e95dcf9607 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/bd24f13b199d41dfbbbc37e95dcf9607 2023-04-06 18:56:42,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f7fba0239cb24054aa8e485657698cf2 2023-04-06 18:56:42,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/a1c5bf3613ea4d8bb5e653e05905fe12 2023-04-06 18:56:42,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/74881cd8570a4a51a002bac19b080e13 2023-04-06 18:56:42,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/50292605e68c41c18f37ec0a1cc78941 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/50292605e68c41c18f37ec0a1cc78941 2023-04-06 18:56:42,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/59eac5feea90420cb5c74a9528d3a0be 2023-04-06 18:56:42,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f2f88628aade4cf9bbc6e9e6e5d18ceb 2023-04-06 18:56:42,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7402a864fc83468ab4e01d232ae60db2 2023-04-06 18:56:42,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f408800bb38345ca82c7fc3a6bdd45c7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f408800bb38345ca82c7fc3a6bdd45c7 2023-04-06 18:56:42,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/1e33640c825b428098c79c89aff0be56 2023-04-06 18:56:42,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/d9f16bcdff6444ff8a7dfbf1b0ff5603 2023-04-06 18:56:42,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0977c86e088f47dd8d58e4bb5a3bae8d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/0977c86e088f47dd8d58e4bb5a3bae8d 2023-04-06 18:56:42,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/b9e8708e1b9c4a86830ea269bf6330f1 2023-04-06 18:56:42,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/390dd5b3d87948a2a0a5e831101ac3d7 2023-04-06 18:56:42,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9fb4e84fc8cb4e8eaf0c1697480d6eb3 2023-04-06 18:56:42,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e3151b2b724744bc9a0436cea3dbf387 2023-04-06 18:56:42,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/658bfeca81ca45619558076ac9a43606 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/658bfeca81ca45619558076ac9a43606 2023-04-06 18:56:42,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/7104242603ce4ca3a9885a16cd356f43 2023-04-06 18:56:42,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/2cacb13219cc44be99376eceecf97464 2023-04-06 18:56:42,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/611ba3f7b0f54361afeceff1144f750e 2023-04-06 18:56:42,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/recovered.edits/605.seqid, newMaxSeqId=605, maxSeqId=1 2023-04-06 18:56:42,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8. 2023-04-06 18:56:42,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 97c1cc7a8c861764bde589e4ad2af6a8: 2023-04-06 18:56:42,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:42,909 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=97c1cc7a8c861764bde589e4ad2af6a8, regionState=CLOSED 2023-04-06 18:56:42,909 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807402909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807402909"}]},"ts":"1680807402909"} 2023-04-06 18:56:42,912 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-06 18:56:42,912 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 97c1cc7a8c861764bde589e4ad2af6a8, server=jenkins-hbase20.apache.org,34607,1680807373848 in 1.6260 sec 2023-04-06 18:56:42,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-06 18:56:42,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=97c1cc7a8c861764bde589e4ad2af6a8, UNASSIGN in 1.6330 sec 2023-04-06 18:56:42,915 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807402915"}]},"ts":"1680807402915"} 2023-04-06 18:56:42,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:56:42,917 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:56:42,919 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6500 sec 2023-04-06 18:56:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 18:56:43,386 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-06 18:56:43,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:56:43,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,402 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,403 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 18:56:43,410 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:43,414 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/recovered.edits] 2023-04-06 18:56:43,418 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3853461f74f6446c9cf6506372bef99d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/3853461f74f6446c9cf6506372bef99d 2023-04-06 18:56:43,420 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a49533e84fd941a096bf774261ef4ccf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/a49533e84fd941a096bf774261ef4ccf 2023-04-06 18:56:43,421 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/bc5fb43ee21b488d8ef0cf7205a5aaff to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/A/bc5fb43ee21b488d8ef0cf7205a5aaff 2023-04-06 18:56:43,425 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/dab47be42e4346f6bf1c1b6531ef71a3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/dab47be42e4346f6bf1c1b6531ef71a3 2023-04-06 18:56:43,428 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3cef5955d0c425dbb1c482711cbd73d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/f3cef5955d0c425dbb1c482711cbd73d 2023-04-06 18:56:43,431 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fd48cea0044640baa8d9d0b3a236c48c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/B/fd48cea0044640baa8d9d0b3a236c48c 2023-04-06 18:56:43,435 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9b6452e3155e4dbf90a9ef9e81ea23e3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/9b6452e3155e4dbf90a9ef9e81ea23e3 2023-04-06 18:56:43,438 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e07feed4c2d646e68d5acb4280bbc377 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/e07feed4c2d646e68d5acb4280bbc377 2023-04-06 18:56:43,440 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f95cc11fd3534d62af6eba3e5c9c97e1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/C/f95cc11fd3534d62af6eba3e5c9c97e1 2023-04-06 18:56:43,445 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/recovered.edits/605.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8/recovered.edits/605.seqid 2023-04-06 18:56:43,446 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/97c1cc7a8c861764bde589e4ad2af6a8 2023-04-06 18:56:43,446 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:56:43,450 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,472 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:56:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 18:56:43,508 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:56:43,509 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,509 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:56:43,509 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807403509"}]},"ts":"9223372036854775807"} 2023-04-06 18:56:43,512 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:56:43,512 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 97c1cc7a8c861764bde589e4ad2af6a8, NAME => 'TestAcidGuarantees,,1680807377571.97c1cc7a8c861764bde589e4ad2af6a8.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:56:43,512 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:56:43,512 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807403512"}]},"ts":"9223372036854775807"} 2023-04-06 18:56:43,514 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:56:43,515 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,517 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 120 msec 2023-04-06 18:56:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 18:56:43,709 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-06 18:56:43,730 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd95a05-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1723357671_15 at /127.0.0.1:57718 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:34607-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd95a05-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1723357671_15 at /127.0.0.1:53432 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd95a05-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd95a05-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=590 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=421 (was 277) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=3022 (was 3531) 2023-04-06 18:56:43,743 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233, OpenFileDescriptor=590, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=166, AvailableMemoryMB=3022 2023-04-06 18:56:43,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:56:43,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:43,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:43,748 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:56:43,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-06 18:56:43,749 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:56:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:43,751 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:43,752 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 empty. 2023-04-06 18:56:43,752 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:43,752 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:56:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:44,164 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:44,170 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e8550365eba5972a2436e7d067fedd10, NAME => 'TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:56:44,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:44,580 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:44,581 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e8550365eba5972a2436e7d067fedd10, disabling compactions & flushes 2023-04-06 18:56:44,581 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,581 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,581 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. after waiting 0 ms 2023-04-06 18:56:44,581 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,581 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,582 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:44,587 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:56:44,588 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807404588"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807404588"}]},"ts":"1680807404588"} 2023-04-06 18:56:44,590 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:56:44,591 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:56:44,591 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807404591"}]},"ts":"1680807404591"} 2023-04-06 18:56:44,592 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:56:44,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, ASSIGN}] 2023-04-06 18:56:44,596 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, ASSIGN 2023-04-06 18:56:44,597 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:56:44,749 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:44,749 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807404749"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807404749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807404749"}]},"ts":"1680807404749"} 2023-04-06 18:56:44,752 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:44,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:44,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e8550365eba5972a2436e7d067fedd10, NAME => 'TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:44,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:44,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,917 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,918 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A 2023-04-06 18:56:44,919 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A 2023-04-06 18:56:44,919 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:44,919 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName A 2023-04-06 18:56:44,920 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:44,920 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,922 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B 2023-04-06 18:56:44,922 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B 2023-04-06 18:56:44,922 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:44,923 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName B 2023-04-06 18:56:44,923 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:44,923 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,925 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C 2023-04-06 18:56:44,925 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C 2023-04-06 18:56:44,925 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:44,926 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName C 2023-04-06 18:56:44,926 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:44,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:56:44,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:44,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:56:44,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e8550365eba5972a2436e7d067fedd10; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69451629, jitterRate=0.034909918904304504}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:56:44,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:44,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., pid=18, masterSystemTime=1680807404905 2023-04-06 18:56:44,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:44,941 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:44,942 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807404941"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807404941"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807404941"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807404941"}]},"ts":"1680807404941"} 2023-04-06 18:56:44,946 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-06 18:56:44,946 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 in 192 msec 2023-04-06 18:56:44,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-06 18:56:44,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, ASSIGN in 352 msec 2023-04-06 18:56:44,949 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:56:44,949 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807404949"}]},"ts":"1680807404949"} 2023-04-06 18:56:44,951 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:56:44,952 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:56:44,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2060 sec 2023-04-06 18:56:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 18:56:45,858 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-06 18:56:45,860 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x65f279d2 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:45,867 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@556231fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:45,871 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:45,878 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:45,880 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:56:45,884 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:37100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:56:45,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:56:45,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:56:45,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 18:56:46,313 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:56:46,315 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:56:46,316 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:56:46,322 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 18:56:46,335 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, REOPEN/MOVE}] 2023-04-06 18:56:46,336 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, REOPEN/MOVE 2023-04-06 18:56:46,337 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,337 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807406337"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807406337"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807406337"}]},"ts":"1680807406337"} 2023-04-06 18:56:46,338 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:46,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e8550365eba5972a2436e7d067fedd10, disabling compactions & flushes 2023-04-06 18:56:46,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. after waiting 0 ms 2023-04-06 18:56:46,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 18:56:46,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:46,501 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e8550365eba5972a2436e7d067fedd10 to self. 2023-04-06 18:56:46,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,505 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=CLOSED 2023-04-06 18:56:46,505 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807406505"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807406505"}]},"ts":"1680807406505"} 2023-04-06 18:56:46,510 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-06 18:56:46,510 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 in 170 msec 2023-04-06 18:56:46,511 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=true 2023-04-06 18:56:46,661 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,662 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807406661"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807406661"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807406661"}]},"ts":"1680807406661"} 2023-04-06 18:56:46,665 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:56:46,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e8550365eba5972a2436e7d067fedd10, NAME => 'TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:56:46,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:56:46,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,832 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,834 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A 2023-04-06 18:56:46,834 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A 2023-04-06 18:56:46,834 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:46,838 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName A 2023-04-06 18:56:46,841 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:46,841 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,842 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B 2023-04-06 18:56:46,842 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B 2023-04-06 18:56:46,842 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:46,842 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName B 2023-04-06 18:56:46,843 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:46,843 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,844 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C 2023-04-06 18:56:46,844 DEBUG [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C 2023-04-06 18:56:46,844 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:56:46,844 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e8550365eba5972a2436e7d067fedd10 columnFamilyName C 2023-04-06 18:56:46,845 INFO [StoreOpener-e8550365eba5972a2436e7d067fedd10-1] regionserver.HStore(310): Store=e8550365eba5972a2436e7d067fedd10/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:56:46,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:56:46,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened e8550365eba5972a2436e7d067fedd10; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74378613, jitterRate=0.1083277016878128}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:56:46,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:46,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., pid=23, masterSystemTime=1680807406818 2023-04-06 18:56:46,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,853 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,853 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807406853"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807406853"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807406853"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807406853"}]},"ts":"1680807406853"} 2023-04-06 18:56:46,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-06 18:56:46,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 in 189 msec 2023-04-06 18:56:46,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-06 18:56:46,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, REOPEN/MOVE in 521 msec 2023-04-06 18:56:46,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-06 18:56:46,862 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 540 msec 2023-04-06 18:56:46,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 968 msec 2023-04-06 18:56:46,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-06 18:56:46,872 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x3b75e45b to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,877 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d781def, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,877 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6017755a to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,881 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@595355a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,881 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x633c81ba to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,886 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38da9522, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,886 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x661b21c2 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,892 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a84dba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,892 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x43a14bcf to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,896 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b7748, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,896 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x2c411e08 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,901 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bd79dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,902 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7605c204 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,918 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52902392, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,918 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x16505096 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,928 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e1da5ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,929 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x03e1990e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:56:46,936 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a1bd04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:56:46,945 DEBUG [hconnection-0x2a9baa5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,946 DEBUG [hconnection-0x14f1abe7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,946 DEBUG [hconnection-0x373c062d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,948 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,948 DEBUG [hconnection-0x13ca27c6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,949 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,959 DEBUG [hconnection-0x1401f50f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,959 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,959 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,960 DEBUG [hconnection-0x7658fa5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,959 DEBUG [hconnection-0x5ae2e9a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,960 DEBUG [hconnection-0x16d968c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,962 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,960 DEBUG [hconnection-0x1a366cf8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:56:46,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:46,970 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,970 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,970 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,973 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:56:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:46,978 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:46,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:46,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:46,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:46,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:46,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:46,980 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:46,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:46,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:46,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:46,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:46,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:46,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:46,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:46,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:46,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:46,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:46,987 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:46,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:46,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:46,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:46,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:46,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:46,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:46,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:46,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:46,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:46,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:46,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3218503a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:46,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:46,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,001 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:47,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:47,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:47,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:47,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:47,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:47,005 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:47,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:47,006 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:47,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:47,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:47,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:47,006 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:47,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:47,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:47,009 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,019 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:47,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:47,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:47,020 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:47,020 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,020 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:47,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:47,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807407015, max timestamp=1680807407021Num uniques 859; 2023-04-06 18:56:47,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:47,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807407015, max timestamp=1680807407021Num uniques 859; 2023-04-06 18:56:47,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:47,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807407015, max timestamp=1680807407021Num uniques 859; 2023-04-06 18:56:47,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:47,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807467041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807467042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807467045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807467046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,069 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040672c1fe5a23b2486e9e4df1c2e0460ff0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672c1fe5a23b2486e9e4df1c2e0460ff0 2023-04-06 18:56:47,079 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/b736476306714bf8bb04084d237d1fb5 2023-04-06 18:56:47,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1e6c2710b2354d059a3752d0cbecab70 2023-04-06 18:56:47,120 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:47,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:47,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:47,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:47,135 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:47,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:47,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:47,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:47,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:47,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:47,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:47,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:47,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:47,146 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:47,146 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:47,146 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:47,146 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:47,146 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:47,147 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:47,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:47,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:47,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807467146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807467147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807467149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:47,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:47,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,154 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:47,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:47,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568b12b1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:47,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:47,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/67e75ba49cb24a6dbece697832af6306 2023-04-06 18:56:47,157 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,158 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,158 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807467154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:47,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:47,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:47,159 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:47,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:47,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:47,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:47,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/b736476306714bf8bb04084d237d1fb5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5 2023-04-06 18:56:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:47,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:47,173 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:47,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:47,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:47,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:47,173 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:47,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5, entries=150, sequenceid=15, filesize=25.1 K 2023-04-06 18:56:47,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:47,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:47,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:47,175 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:47,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:47,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:47,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:47,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:47,177 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:47,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:47,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,178 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:47,178 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:47,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1e6c2710b2354d059a3752d0cbecab70 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70 2023-04-06 18:56:47,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:47,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:47,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70, entries=150, sequenceid=15, filesize=11.6 K 2023-04-06 18:56:47,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/67e75ba49cb24a6dbece697832af6306 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306 2023-04-06 18:56:47,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306, entries=150, sequenceid=15, filesize=11.6 K 2023-04-06 18:56:47,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e8550365eba5972a2436e7d067fedd10 in 212ms, sequenceid=15, compaction requested=false 2023-04-06 18:56:47,200 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:47,278 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:47,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:47,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:47,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:47,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:47,284 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:47,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:47,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:47,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:47,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,286 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:47,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:47,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:47,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:47,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:47,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:47,287 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:47,287 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:47,287 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:47,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:47,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:47,288 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:47,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:47,289 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:47,289 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:47,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:47,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:47,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807467348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807467349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807467351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807467361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807467652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807467654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807467656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807467665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:47,724 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304067e09f06b525a4d8dbf8e4d49646ab8d1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e09f06b525a4d8dbf8e4d49646ab8d1 2023-04-06 18:56:47,731 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/3d05f996cdee45d8b654c6db9aa79633 2023-04-06 18:56:47,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:47,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807467963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,143 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/28850498cd04481ca067364baa76a90f 2023-04-06 18:56:48,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:48,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807468154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807468158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:48,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807468166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:48,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807468171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,445 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:56:48,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807468467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,571 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/3189ae23ccb84d949d829557bb472e7f 2023-04-06 18:56:48,577 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/3d05f996cdee45d8b654c6db9aa79633 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633 2023-04-06 18:56:48,582 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633, entries=150, sequenceid=52, filesize=25.1 K 2023-04-06 18:56:48,583 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/28850498cd04481ca067364baa76a90f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f 2023-04-06 18:56:48,588 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 18:56:48,589 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/3189ae23ccb84d949d829557bb472e7f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f 2023-04-06 18:56:48,595 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 18:56:48,596 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e8550365eba5972a2436e7d067fedd10 in 1308ms, sequenceid=52, compaction requested=false 2023-04-06 18:56:48,596 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:48,596 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:48,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:48,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397a8745[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:48,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:48,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,603 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:48,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:48,604 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:48,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:48,605 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:48,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:48,607 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:48,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:48,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:48,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:48,607 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:48,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:48,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:48,610 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:48,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:48,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:48,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:48,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:48,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:48,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:48,615 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:48,615 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:48,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:48,715 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:48,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:48,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:48,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:48,726 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:48,726 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:48,726 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:48,726 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:48,726 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,726 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:48,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:48,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:48,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:48,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:48,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:48,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:48,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:48,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:48,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:48,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:48,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:48,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:48,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:48,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:48,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:48,729 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:48,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:48,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:48,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:48,730 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:48,730 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:48,730 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:49,158 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066b12393118174ab4a7ea60f4f6e97e07 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066b12393118174ab4a7ea60f4f6e97e07 2023-04-06 18:56:49,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:49,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,184 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/7db4f847f98f4331996408484d1650c6 2023-04-06 18:56:49,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807409180, max timestamp=1680807409192Num uniques 859; 2023-04-06 18:56:49,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:49,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807409180, max timestamp=1680807409192Num uniques 859; 2023-04-06 18:56:49,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807409180, max timestamp=1680807409192Num uniques 859; 2023-04-06 18:56:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:56:49,217 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/bed645f6ade44ae28d25e95c17494ef9 2023-04-06 18:56:49,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807469215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807469217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807469218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807469224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,235 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/30498960341441dabd441007b3b699dc 2023-04-06 18:56:49,243 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/7db4f847f98f4331996408484d1650c6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6 2023-04-06 18:56:49,249 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6, entries=150, sequenceid=62, filesize=25.1 K 2023-04-06 18:56:49,250 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/bed645f6ade44ae28d25e95c17494ef9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9 2023-04-06 18:56:49,256 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 18:56:49,258 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/30498960341441dabd441007b3b699dc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc 2023-04-06 18:56:49,262 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 18:56:49,263 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e8550365eba5972a2436e7d067fedd10 in 534ms, sequenceid=62, compaction requested=true 2023-04-06 18:56:49,263 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:49,263 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5280e77e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,267 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,270 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,270 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,272 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:49,275 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,275 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:49,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=140.89 KB heapSize=400.45 KB 2023-04-06 18:56:49,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807409208, max timestamp=1680807409323Num uniques 1068; 2023-04-06 18:56:49,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:56:49,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807409208, max timestamp=1680807409323Num uniques 1068; 2023-04-06 18:56:49,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:56:49,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807409208, max timestamp=1680807409323Num uniques 1068; 2023-04-06 18:56:49,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:49,326 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:49,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680807409338, max timestamp=1680807409347Num uniques 859; 2023-04-06 18:56:49,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 18:56:49,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680807409338, max timestamp=1680807409347Num uniques 859; 2023-04-06 18:56:49,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 18:56:49,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680807409338, max timestamp=1680807409347Num uniques 859; 2023-04-06 18:56:49,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 18:56:49,350 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062c670cc3b96b413d8f7cde2ad9d23a94 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062c670cc3b96b413d8f7cde2ad9d23a94 2023-04-06 18:56:49,357 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/af7e1a6e2d9c4aa491bc587cbb776c86 2023-04-06 18:56:49,375 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807469372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807469373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1680807409362, max timestamp=1680807409373Num uniques 1168; 2023-04-06 18:56:49,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53480, new segment data size=7680, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-06 18:56:49,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7686, totalHeapSize=17712, min timestamp=1680807409362, max timestamp=1680807409373Num uniques 1168; 2023-04-06 18:56:49,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53486, new segment data size=7686, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-06 18:56:49,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807409362, max timestamp=1680807409373Num uniques 1168; 2023-04-06 18:56:49,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-06 18:56:49,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,383 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,390 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,390 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,390 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,390 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e9f494[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,396 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,398 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,398 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,403 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,403 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807409389, max timestamp=1680807409412Num uniques 1027; 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807409389, max timestamp=1680807409412Num uniques 1027; 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807409389, max timestamp=1680807409412Num uniques 1027; 2023-04-06 18:56:49,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:56:49,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807469441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680807409430, max timestamp=1680807409440Num uniques 1086; 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680807409430, max timestamp=1680807409440Num uniques 1086; 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680807409430, max timestamp=1680807409440Num uniques 1086; 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-06 18:56:49,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-06 18:56:49,503 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,508 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,511 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,511 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,511 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,511 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7964e2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680807409469, max timestamp=1680807409515Num uniques 1195; 2023-04-06 18:56:49,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-06 18:56:49,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680807409469, max timestamp=1680807409515Num uniques 1195; 2023-04-06 18:56:49,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-06 18:56:49,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680807409469, max timestamp=1680807409515Num uniques 1195; 2023-04-06 18:56:49,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-06 18:56:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,519 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,519 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,526 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,526 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,526 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,528 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,534 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,534 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,534 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807469533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807469535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807469535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807469536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807469543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,634 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,637 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807469639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,641 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,641 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,642 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807469641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807469641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,643 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807469642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41548e87[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,645 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,645 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,648 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,648 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,655 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,655 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807469745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,755 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,758 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,759 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,760 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,760 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,760 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ba2ac4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,764 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,766 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,766 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,767 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,767 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,767 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,767 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/575d5d2e74a34a6190593408cd863873 2023-04-06 18:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,771 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,771 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,771 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807469843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807469844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807469845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807469845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,871 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,874 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,875 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,877 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,877 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,877 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,877 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,877 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:49,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791a3193[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:49,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:49,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,881 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:49,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:49,882 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:49,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:49,885 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:49,885 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:49,885 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:49,885 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:49,885 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:49,886 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,886 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,886 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,890 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,891 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:49,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:49,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:49,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:49,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,892 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:49,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:49,892 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:49,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:49,992 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:49,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:49,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:49,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:49,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:49,995 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:49,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:49,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:49,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:49,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:49,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:49,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:49,997 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,997 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:49,998 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:49,998 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:49,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:49,999 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:49,999 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:50,000 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:50,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea30426[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:50,000 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,001 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:50,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:50,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:50,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:50,002 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:50,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:50,004 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:50,004 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:50,004 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:50,004 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:50,004 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:50,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:50,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,009 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:50,008 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,009 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:50,009 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,009 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,009 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807470048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,109 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:50,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:50,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:50,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:50,112 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:50,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:50,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:50,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:50,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:50,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:50,115 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,115 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:50,115 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:50,115 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:50,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:50,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,117 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,118 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,118 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:50,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6691c3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:50,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:50,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,119 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:50,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:50,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:50,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:50,121 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:50,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:50,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:50,123 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:50,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:50,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:50,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:50,123 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:50,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,125 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:50,136 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:50,136 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:50,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:50,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807470147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807470147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807470147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807470148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/94e00307b3824c9d927a8bd86173e756 2023-04-06 18:56:50,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/af7e1a6e2d9c4aa491bc587cbb776c86 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86 2023-04-06 18:56:50,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86, entries=200, sequenceid=100, filesize=31.9 K 2023-04-06 18:56:50,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/575d5d2e74a34a6190593408cd863873 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873 2023-04-06 18:56:50,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873, entries=150, sequenceid=100, filesize=11.6 K 2023-04-06 18:56:50,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/94e00307b3824c9d927a8bd86173e756 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756 2023-04-06 18:56:50,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756, entries=150, sequenceid=100, filesize=11.6 K 2023-04-06 18:56:50,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=161.02 KB/164880 for e8550365eba5972a2436e7d067fedd10 in 873ms, sequenceid=100, compaction requested=true 2023-04-06 18:56:50,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:50,197 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:50,197 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:50,198 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:50,198 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:50,198 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:56:50,198 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:56:50,198 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,198 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,198 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=46.4 K 2023-04-06 18:56:50,198 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=107.3 K 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1e6c2710b2354d059a3752d0cbecab70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680807406970 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b736476306714bf8bb04084d237d1fb5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680807406970 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 28850498cd04481ca067364baa76a90f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807407033 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 3d05f996cdee45d8b654c6db9aa79633, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807407033 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting bed645f6ade44ae28d25e95c17494ef9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807407336 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7db4f847f98f4331996408484d1650c6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807407336 2023-04-06 18:56:50,199 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 575d5d2e74a34a6190593408cd863873, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:50,200 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting af7e1a6e2d9c4aa491bc587cbb776c86, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:50,207 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:50,214 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#132 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:50,235 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/40a75bbbbe8944ed91963609ab6dac1e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/40a75bbbbe8944ed91963609ab6dac1e 2023-04-06 18:56:50,236 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:50,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:50,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:50,244 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 40a75bbbbe8944ed91963609ab6dac1e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:50,244 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:50,244 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=12, startTime=1680807410197; duration=0sec 2023-04-06 18:56:50,244 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:50,245 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:50,246 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:50,246 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:56:50,246 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,246 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=46.4 K 2023-04-06 18:56:50,247 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 67e75ba49cb24a6dbece697832af6306, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680807406970 2023-04-06 18:56:50,247 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3189ae23ccb84d949d829557bb472e7f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807407033 2023-04-06 18:56:50,248 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 30498960341441dabd441007b3b699dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807407336 2023-04-06 18:56:50,248 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 94e00307b3824c9d927a8bd86173e756, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:50,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:50,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:50,262 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:50,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:50,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:50,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,264 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:50,268 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:50,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:50,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:50,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:50,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:50,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:50,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:50,269 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,269 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:50,269 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=161.02 KB heapSize=442.55 KB 2023-04-06 18:56:50,269 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:50,269 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:50,269 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:50,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:50,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:50,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:50,280 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/2bc71ae6a602430c88f697213d278761 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2bc71ae6a602430c88f697213d278761 2023-04-06 18:56:50,294 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 2bc71ae6a602430c88f697213d278761(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:50,294 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:50,294 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=12, startTime=1680807410197; duration=0sec 2023-04-06 18:56:50,294 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:50,313 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c31750ea44a14aff8dc9882415ce1bb4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c31750ea44a14aff8dc9882415ce1bb4 2023-04-06 18:56:50,320 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/9a543d7f87444b7691efbedafbab69fd 2023-04-06 18:56:50,334 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/cf7f5f0b15ce407b977593120689f15d 2023-04-06 18:56:50,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:50,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:50,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807470567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807470651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807470653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807470653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807470654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:50,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807470670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,755 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/72aec155eaf54ba1b3f523dcc223f9ed 2023-04-06 18:56:50,759 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/9a543d7f87444b7691efbedafbab69fd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd 2023-04-06 18:56:50,766 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd, entries=150, sequenceid=206, filesize=25.3 K 2023-04-06 18:56:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/cf7f5f0b15ce407b977593120689f15d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d 2023-04-06 18:56:50,771 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d, entries=150, sequenceid=206, filesize=11.8 K 2023-04-06 18:56:50,771 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/72aec155eaf54ba1b3f523dcc223f9ed as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed 2023-04-06 18:56:50,775 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed, entries=150, sequenceid=206, filesize=11.8 K 2023-04-06 18:56:50,776 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~442.50 KB/453120, currentSize=26.84 KB/27480 for e8550365eba5972a2436e7d067fedd10 in 507ms, sequenceid=206, compaction requested=false 2023-04-06 18:56:50,776 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:50,776 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:50,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:50,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:50,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:50,777 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:50,778 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:50,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4f6861[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:50,785 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:50,785 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,792 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:50,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:50,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:50,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:50,793 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:50,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,793 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:50,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:50,797 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:50,797 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:50,797 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:50,797 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:50,797 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:50,801 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,801 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,801 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:50,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,817 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:50,817 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:50,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:50,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:50,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:50,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:50,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:50,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:50,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:50,906 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040654bea46f354446c2870bff1e6c2d059a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040654bea46f354446c2870bff1e6c2d059a 2023-04-06 18:56:50,917 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:50,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:50,924 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:50,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:50,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:50,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,926 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:50,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:50,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:50,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:50,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:50,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:50,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:50,938 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:50,938 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:50,938 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:50,938 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:50,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:50,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:50,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f15092[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:50,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:50,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:50,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807410908, max timestamp=1680807410955Num uniques 859; 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807410908, max timestamp=1680807410955Num uniques 859; 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807410908, max timestamp=1680807410955Num uniques 859; 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:50,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:50,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:50,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:50,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:50,961 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:50,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:50,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:50,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:50,962 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:50,968 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,968 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,968 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:50,968 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:50,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:50,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:50,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:50,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:50,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:50,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:50,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:50,976 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:50,976 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:50,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807471018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,061 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/9570124a403d46c783fb7910974bb081 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9570124a403d46c783fb7910974bb081 2023-04-06 18:56:51,066 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 9570124a403d46c783fb7910974bb081(size=25.3 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:51,066 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:51,066 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=12, startTime=1680807410197; duration=0sec 2023-04-06 18:56:51,066 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:51,076 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,079 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1075e45e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,090 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,090 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,094 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:51,094 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,094 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807471123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,194 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,197 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,203 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,203 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f52871a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,204 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,204 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,206 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,208 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,209 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,209 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,209 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,209 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,209 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,209 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,210 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,210 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,214 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,214 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,214 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,214 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,314 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,316 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,317 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,319 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,319 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,319 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,319 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,320 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1bf0f1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,323 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,324 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,324 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:51,329 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,329 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807471326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,333 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/2a3f6fc016964dd7b7869e3494321907 2023-04-06 18:56:51,429 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,432 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,433 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,435 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f256e0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,437 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,437 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,441 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,441 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,443 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,446 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,446 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,546 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,549 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,551 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,552 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391a1574[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,555 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,555 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,555 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,555 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,555 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,557 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,557 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,558 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,559 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:51,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,562 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,562 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807471633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807471657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,662 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807471662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,664 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,666 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,666 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807471665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807471665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,677 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@723b3f1a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,690 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,690 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,690 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,692 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,697 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,697 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,698 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,698 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,698 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,711 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,711 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1d8bc5dff68c4390b98d0da809fa42e5 2023-04-06 18:56:51,812 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,815 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,819 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e2c5d4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,822 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,823 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,824 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,825 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,829 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,829 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,929 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:51,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:51,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:51,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:51,932 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:51,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:51,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:51,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:51,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:51,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:51,935 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,935 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:51,935 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:51,935 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:51,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:51,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2333c09e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:51,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:51,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:51,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:51,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:51,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:51,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,940 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:51,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:51,942 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:51,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:51,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:51,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:51,942 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:51,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:51,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:51,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:51,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:51,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:51,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:51,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:51,946 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:51,946 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:51,946 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,046 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:52,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:52,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:52,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:52,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:52,050 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:52,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:52,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:52,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:52,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:52,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:52,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:52,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:52,052 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,052 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:52,052 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:52,052 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:52,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,053 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:52,054 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:52,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aebc039[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:52,054 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,055 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:52,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:52,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:52,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:52,055 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:52,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:52,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:52,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:52,057 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:52,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:52,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:52,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:52,058 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:52,058 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,058 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,058 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:52,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:52,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:52,061 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:52,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,061 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:52,062 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:52,062 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807472136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,162 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:52,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:52,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:52,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:52,164 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:52,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:52,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:52,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/03d5f53593d245d7af2178e5529d6f48 2023-04-06 18:56:52,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,168 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:52,168 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:52,168 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,168 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:52,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:52,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:52,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:52,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:52,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:52,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:52,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:52,169 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,169 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:52,169 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:52,169 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:52,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/2a3f6fc016964dd7b7869e3494321907 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907 2023-04-06 18:56:52,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:52,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6103e591[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:52,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:52,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:52,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:52,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907, entries=250, sequenceid=217, filesize=38.9 K 2023-04-06 18:56:52,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1d8bc5dff68c4390b98d0da809fa42e5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5 2023-04-06 18:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:52,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:52,178 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:52,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:52,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:52,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:52,179 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:52,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5, entries=150, sequenceid=217, filesize=11.8 K 2023-04-06 18:56:52,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/03d5f53593d245d7af2178e5529d6f48 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48 2023-04-06 18:56:52,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:52,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:52,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48, entries=150, sequenceid=217, filesize=11.8 K 2023-04-06 18:56:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:52,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e8550365eba5972a2436e7d067fedd10 in 1301ms, sequenceid=217, compaction requested=true 2023-04-06 18:56:52,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:52,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:52,185 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:52,185 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:52,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:52,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:52,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:52,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:52,186 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:52,186 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:52,186 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:52,186 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:56:52,186 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,186 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/40a75bbbbe8944ed91963609ab6dac1e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=35.3 K 2023-04-06 18:56:52,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:52,187 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:52,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:52,187 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:56:52,187 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:52,187 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9570124a403d46c783fb7910974bb081, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=89.5 K 2023-04-06 18:56:52,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:52,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:52,186 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:52,187 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 40a75bbbbe8944ed91963609ab6dac1e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:52,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:52,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,187 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9570124a403d46c783fb7910974bb081, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:52,188 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting cf7f5f0b15ce407b977593120689f15d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680807409527 2023-04-06 18:56:52,188 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9a543d7f87444b7691efbedafbab69fd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680807409527 2023-04-06 18:56:52,188 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1d8bc5dff68c4390b98d0da809fa42e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410875 2023-04-06 18:56:52,188 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2a3f6fc016964dd7b7869e3494321907, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410564 2023-04-06 18:56:52,193 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#140 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:52,206 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#141 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:52,286 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:52,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:52,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:52,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:52,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:52,289 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:52,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:52,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:52,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,290 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:52,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:52,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:52,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:52,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:52,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:52,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:52,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:52,291 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:52,292 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:52,609 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1ed660b8699e429d949fe68ea8375bf5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ed660b8699e429d949fe68ea8375bf5 2023-04-06 18:56:52,613 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 1ed660b8699e429d949fe68ea8375bf5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:52,614 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:52,614 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=13, startTime=1680807412185; duration=0sec 2023-04-06 18:56:52,614 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:52,614 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:52,615 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:52,615 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:56:52,615 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:52,615 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2bc71ae6a602430c88f697213d278761, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=35.3 K 2023-04-06 18:56:52,615 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2bc71ae6a602430c88f697213d278761, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680807409215 2023-04-06 18:56:52,615 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 72aec155eaf54ba1b3f523dcc223f9ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1680807409527 2023-04-06 18:56:52,616 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 03d5f53593d245d7af2178e5529d6f48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410875 2023-04-06 18:56:52,622 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:52,638 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/201a412165b7476f92fbcb78ab50e7bb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/201a412165b7476f92fbcb78ab50e7bb 2023-04-06 18:56:52,642 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 201a412165b7476f92fbcb78ab50e7bb(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:52,642 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:52,642 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=13, startTime=1680807412185; duration=0sec 2023-04-06 18:56:52,642 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:52,705 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d117096a4165417eaeb9cfa5609e08d0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d117096a4165417eaeb9cfa5609e08d0 2023-04-06 18:56:52,710 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/bbd0cb38e0944300a264b024b22ba16d 2023-04-06 18:56:53,038 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/4a4a03a7478b415387918bdd9c63b5b9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/4a4a03a7478b415387918bdd9c63b5b9 2023-04-06 18:56:53,042 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 4a4a03a7478b415387918bdd9c63b5b9(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:53,042 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:53,042 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=13, startTime=1680807412185; duration=0sec 2023-04-06 18:56:53,042 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:53,127 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/600478d784e247a298e06eaf4700a523 2023-04-06 18:56:53,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:53,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:53,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38338 deadline: 1680807473160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807473262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807473465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,560 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/b7f5c6d907234729b5a55ebacb5b316c 2023-04-06 18:56:53,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/bbd0cb38e0944300a264b024b22ba16d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d 2023-04-06 18:56:53,569 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d, entries=150, sequenceid=255, filesize=25.3 K 2023-04-06 18:56:53,570 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/600478d784e247a298e06eaf4700a523 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523 2023-04-06 18:56:53,574 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523, entries=150, sequenceid=255, filesize=11.8 K 2023-04-06 18:56:53,574 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/b7f5c6d907234729b5a55ebacb5b316c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c 2023-04-06 18:56:53,579 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c, entries=150, sequenceid=255, filesize=11.8 K 2023-04-06 18:56:53,580 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e8550365eba5972a2436e7d067fedd10 in 1289ms, sequenceid=255, compaction requested=false 2023-04-06 18:56:53,580 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:53,580 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:53,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:53,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:53,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:53,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,581 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:53,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e4000[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:53,592 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:53,592 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:53,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:53,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:53,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:53,600 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:53,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:53,603 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:53,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:53,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:53,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:53,603 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:53,604 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,604 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:53,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,609 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:53,609 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:53,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:53,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:53,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:53,709 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:53,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:53,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:53,714 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:53,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:53,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:53,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,715 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,715 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:53,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:53,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:53,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:53,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:53,722 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:53,722 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:53,722 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:53,722 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:53,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807413712, max timestamp=1680807413724Num uniques 859; 2023-04-06 18:56:53,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807413712, max timestamp=1680807413724Num uniques 859; 2023-04-06 18:56:53,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:53,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807413712, max timestamp=1680807413724Num uniques 859; 2023-04-06 18:56:53,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:53,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:53,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,731 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:53,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de45fa3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:53,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:53,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:53,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:53,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:53,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:53,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:53,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:53,737 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:53,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:53,739 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:53,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:53,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:53,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:53,739 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:53,740 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,740 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,740 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,741 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:53,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,743 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,744 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:53,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807473743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807473745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807473746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807473746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807473771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,844 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:53,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:53,846 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:53,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:53,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:53,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:53,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807473846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:53,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:53,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:53,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:53,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:53,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:53,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807473851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,852 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807473851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:53,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807473851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:53,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691ab22e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:53,853 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:53,853 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:53,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:53,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:53,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:53,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:53,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:53,856 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:53,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:53,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:53,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:53,856 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:53,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:53,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:53,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:53,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,861 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:53,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,861 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:53,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,961 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:53,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:53,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:53,964 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:53,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:53,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:53,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:53,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:53,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:53,966 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,966 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:53,966 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:53,966 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:53,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:53,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:53,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:53,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:53,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:53,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:53,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174bfdf1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:53,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:53,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:53,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:53,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:53,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:53,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:53,972 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:53,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:53,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:53,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:53,972 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:53,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:53,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:53,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:53,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:53,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:53,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:53,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:53,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:53,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:53,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:53,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:53,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:53,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:53,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:53,977 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:53,977 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:54,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807474049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807474053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807474053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807474054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,077 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:54,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:54,080 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:54,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:54,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:54,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:54,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:54,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:54,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:54,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:54,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:54,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:54,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:54,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:54,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:54,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:54,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:54,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:54,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:54,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2597d158[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:54,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:54,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:54,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:54,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:54,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:54,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:54,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:54,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:54,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:54,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:54,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:54,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,092 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040641c0335ea1f7459dbed35b5143a4bc9a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040641c0335ea1f7459dbed35b5143a4bc9a 2023-04-06 18:56:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:54,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,095 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,095 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,195 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:54,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:54,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:54,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:54,199 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:54,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:54,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:54,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,200 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:54,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:54,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:54,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:54,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:54,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:54,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:54,204 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:54,204 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:54,205 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:54,205 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:54,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:54,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:54,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:54,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:54,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:54,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad19b0e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:54,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:54,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:54,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:54,221 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:54,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:54,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:54,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:54,221 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,224 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:54,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,226 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:54,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,226 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,226 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:54,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807474279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,326 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:54,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:54,329 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:54,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:54,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:54,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:54,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:54,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:54,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:54,338 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,338 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:54,338 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:54,338 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:54,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:54,342 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,342 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:54,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e6cdb8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:54,344 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:54,344 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:54,348 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:54,348 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:54,348 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:54,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:54,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:54,351 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807474353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:54,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:54,354 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:54,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:54,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:54,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:54,354 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:54,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,357 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807474354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807474357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807474355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,362 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:54,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:54,362 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:54,362 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,463 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:54,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:54,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:54,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:54,467 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:54,467 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,467 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:54,467 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:54,467 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:54,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:54,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:54,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,471 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:54,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:54,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:54,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:54,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:54,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:54,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0a14e0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:54,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:54,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,484 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:54,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:54,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:54,484 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,484 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:54,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:54,487 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:54,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:54,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:54,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:54,487 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:54,488 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,488 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,488 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:54,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:54,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:54,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:54,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:54,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:54,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:54,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:54,492 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:54,492 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:54,499 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/2aa2d00366014b689db3cea19a8eb938 2023-04-06 18:56:54,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/f6376b59f6d841df886811642cbff3eb 2023-04-06 18:56:54,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/e8d219148dcc481095345f954d566822 2023-04-06 18:56:54,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/2aa2d00366014b689db3cea19a8eb938 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938 2023-04-06 18:56:54,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938, entries=150, sequenceid=269, filesize=25.4 K 2023-04-06 18:56:54,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/f6376b59f6d841df886811642cbff3eb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb 2023-04-06 18:56:54,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb, entries=100, sequenceid=269, filesize=9.5 K 2023-04-06 18:56:54,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/e8d219148dcc481095345f954d566822 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822 2023-04-06 18:56:54,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822, entries=100, sequenceid=269, filesize=9.5 K 2023-04-06 18:56:54,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e8550365eba5972a2436e7d067fedd10 in 882ms, sequenceid=269, compaction requested=true 2023-04-06 18:56:54,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:54,559 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:54,560 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:54,560 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78074 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:54,561 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:56:54,561 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,561 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/201a412165b7476f92fbcb78ab50e7bb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=76.2 K 2023-04-06 18:56:54,561 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:54,561 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:56:54,561 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,561 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 201a412165b7476f92fbcb78ab50e7bb, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410875 2023-04-06 18:56:54,561 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ed660b8699e429d949fe68ea8375bf5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=33.3 K 2023-04-06 18:56:54,562 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting bbd0cb38e0944300a264b024b22ba16d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680807410979 2023-04-06 18:56:54,562 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1ed660b8699e429d949fe68ea8375bf5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410875 2023-04-06 18:56:54,562 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2aa2d00366014b689db3cea19a8eb938, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807413153 2023-04-06 18:56:54,562 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 600478d784e247a298e06eaf4700a523, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680807410979 2023-04-06 18:56:54,562 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f6376b59f6d841df886811642cbff3eb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807413153 2023-04-06 18:56:54,571 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#149 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:54,585 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#150 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:54,592 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:54,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:54,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:54,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:54,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:54,598 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:54,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:54,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:54,599 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,599 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,599 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:54,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:54,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:54,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:54,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:54,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:54,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:54,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:54,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:54,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:54,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:54,601 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:54,601 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:54,602 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:54,602 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:54,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:54,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807474866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807474866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807474866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807474867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807474968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807474968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807474968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807474971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:54,989 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ad7368e13e4a43b1939a8263b374fe2c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad7368e13e4a43b1939a8263b374fe2c 2023-04-06 18:56:54,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into ad7368e13e4a43b1939a8263b374fe2c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:54,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:54,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=13, startTime=1680807414560; duration=0sec 2023-04-06 18:56:54,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:54,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:54,995 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:54,995 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:56:54,995 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:54,995 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/4a4a03a7478b415387918bdd9c63b5b9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=33.3 K 2023-04-06 18:56:54,996 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4a4a03a7478b415387918bdd9c63b5b9, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1680807410875 2023-04-06 18:56:54,996 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b7f5c6d907234729b5a55ebacb5b316c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680807410979 2023-04-06 18:56:54,996 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e8d219148dcc481095345f954d566822, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807413153 2023-04-06 18:56:55,002 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#152 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:55,024 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/8e0c8cef5e8e4fcd94060fd8113b6565 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8e0c8cef5e8e4fcd94060fd8113b6565 2023-04-06 18:56:55,029 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 8e0c8cef5e8e4fcd94060fd8113b6565(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:55,029 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:55,029 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=13, startTime=1680807414559; duration=0sec 2023-04-06 18:56:55,029 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:55,036 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069434aacd2b374b13b952322121c2cc59 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069434aacd2b374b13b952322121c2cc59 2023-04-06 18:56:55,047 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/8b43c45393f54d50afefe204c55926f5 2023-04-06 18:56:55,061 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/923ff00d89b64a5ca9e025ef16ba260c 2023-04-06 18:56:55,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807475170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807475170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807475171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807475174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807475285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,414 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/a416bd3b3abb44fda8eb8500fd012619 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/a416bd3b3abb44fda8eb8500fd012619 2023-04-06 18:56:55,418 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into a416bd3b3abb44fda8eb8500fd012619(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:55,418 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:55,418 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=13, startTime=1680807414560; duration=0sec 2023-04-06 18:56:55,418 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807475473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807475474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,476 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/62d42d775c9f4f508c7c4073f58fef67 2023-04-06 18:56:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807475475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,481 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/8b43c45393f54d50afefe204c55926f5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5 2023-04-06 18:56:55,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807475480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,489 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5, entries=150, sequenceid=306, filesize=25.4 K 2023-04-06 18:56:55,490 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/923ff00d89b64a5ca9e025ef16ba260c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c 2023-04-06 18:56:55,495 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 18:56:55,496 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/62d42d775c9f4f508c7c4073f58fef67 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67 2023-04-06 18:56:55,500 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67, entries=150, sequenceid=306, filesize=11.9 K 2023-04-06 18:56:55,501 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e8550365eba5972a2436e7d067fedd10 in 899ms, sequenceid=306, compaction requested=false 2023-04-06 18:56:55,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:55,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:55,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:55,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:55,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:55,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:55,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:55,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:55,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:55,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150c1968[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:55,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:55,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:55,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:55,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:55,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:55,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:55,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:55,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:55,511 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:55,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:55,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:55,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:55,511 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:55,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:55,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:55,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:55,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:55,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:55,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:55,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:55,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:55,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:55,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:55,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:55,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,517 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:55,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,517 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:55,617 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:55,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:55,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:55,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:55,620 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:55,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:55,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:55,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:55,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:55,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,621 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:55,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:55,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:55,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:55,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:55,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:55,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:55,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:55,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:55,623 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:55,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:55,639 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ec45e7aa01b94482a62a9474acc380b7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ec45e7aa01b94482a62a9474acc380b7 2023-04-06 18:56:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:55,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:55,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:55,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807415990, max timestamp=1680807415992Num uniques 859; 2023-04-06 18:56:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807475993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:55,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807415990, max timestamp=1680807415992Num uniques 859; 2023-04-06 18:56:55,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680807415989, max timestamp=1680807415992Num uniques 859; 2023-04-06 18:56:55,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:55,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:56:55,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 18:56:56,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807476012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807476014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807476016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,045 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/38aaccd21bc344c6ab4d96b5658c83ee 2023-04-06 18:56:56,054 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ad75aa486a2145b59a6a4defca31477e 2023-04-06 18:56:56,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807476096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807476117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807476117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807476119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807476300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38312 deadline: 1680807476318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807476320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807476325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,473 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/005d84cca8764ab08c73d6ce9de38a3c 2023-04-06 18:56:56,476 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/38aaccd21bc344c6ab4d96b5658c83ee as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee 2023-04-06 18:56:56,480 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee, entries=150, sequenceid=319, filesize=25.4 K 2023-04-06 18:56:56,481 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ad75aa486a2145b59a6a4defca31477e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e 2023-04-06 18:56:56,484 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e, entries=150, sequenceid=319, filesize=11.9 K 2023-04-06 18:56:56,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/005d84cca8764ab08c73d6ce9de38a3c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c 2023-04-06 18:56:56,488 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c, entries=150, sequenceid=319, filesize=11.9 K 2023-04-06 18:56:56,489 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e8550365eba5972a2436e7d067fedd10 in 866ms, sequenceid=319, compaction requested=true 2023-04-06 18:56:56,489 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:56,489 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:56,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:56,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:56,489 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:56,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,490 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:56,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6373f92c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:56,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:56,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:56,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:56,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:56,492 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:56,493 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:56,493 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,493 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,493 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:56,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:56,495 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:56,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:56,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:56,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:56,495 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:56,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:56,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:56,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:56,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,500 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:56,500 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,500 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:56,500 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,600 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:56,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:56,602 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:56,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:56,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:56,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,603 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,604 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:56,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:56,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:56,605 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:56,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:56,605 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:56,605 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:56,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:56,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:56,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:56,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:56,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,607 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:56,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138873f7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:56,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:56,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,610 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:56,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:56,611 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,612 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,612 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:56,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:56,614 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:56,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:56,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:56,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:56,614 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:56,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:56,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,619 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,619 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:56,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807476621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807476621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807476624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807476629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,719 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:56,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:56,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:56,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:56,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:56,722 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:56,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:56,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:56,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,723 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:56,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:56,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:56,725 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,725 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:56,725 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:56,725 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:56,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:56,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,726 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807476724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:56,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7bc872[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:56,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:56,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:56,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:56,729 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:56,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:56,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:56,731 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:56,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:56,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:56,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:56,731 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:56,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,733 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:56,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:56,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:56,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,736 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,736 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:56,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,836 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:56,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:56,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:56,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:56,839 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:56,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:56,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:56,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:56,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:56,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:56,843 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,843 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:56,843 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:56,843 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:56,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:56,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,845 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,845 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:56,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:56,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c95fec[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:56,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:56,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:56,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:56,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:56,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:56,851 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:56,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:56,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:56,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:56,851 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:56,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:56,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:56,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:56,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,868 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,868 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,868 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,868 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:56,868 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:56,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807476928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,968 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:56,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:56,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:56,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:56,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:56,971 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:56,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:56,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:56,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:56,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:56,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:56,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:56,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:56,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:56,979 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,979 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:56,979 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:56,979 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:56,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:56,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448f7b40[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,990 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:56,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:56,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:56,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:56,990 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,991 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:56,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:56,992 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:56,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:56,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:56,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:56,992 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:56,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:56,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:56,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:56,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:56,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:56,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:56,996 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:56,996 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,018 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068e304ebae24f496face49d6c4175060b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068e304ebae24f496face49d6c4175060b 2023-04-06 18:56:57,097 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,099 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,101 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c8bfed[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,107 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,107 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:57,124 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,126 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,126 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807477127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807477129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807477135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,226 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,229 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763cd5e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,234 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,234 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,235 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807477233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,236 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,236 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,237 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,237 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,239 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,239 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,240 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,240 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,240 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,241 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,242 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,245 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,245 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807477291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,295 DEBUG [Thread-935] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:56:57,345 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,348 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,349 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,351 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,352 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7527c461[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,353 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,353 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,353 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,353 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,354 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,356 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,356 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,357 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,357 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,357 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,357 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,360 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,360 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,423 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ac3f9aba41c045099b2a12307acee040 2023-04-06 18:56:57,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/3579c2ea9cad4693b6ac454533a5ffbf 2023-04-06 18:56:57,460 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,462 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,464 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,464 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,464 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,465 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,465 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,465 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,465 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,465 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,466 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,466 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,466 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,466 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,466 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,467 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,467 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4acde4fe[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,468 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,470 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,471 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,472 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,472 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,472 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,472 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,473 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,478 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,478 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,478 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,478 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,478 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,479 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,479 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:57,579 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,582 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,583 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,583 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,583 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,583 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,584 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,584 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,584 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,585 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,585 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,585 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,585 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,585 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,585 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,585 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,586 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b2b1d9d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,593 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,593 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,602 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,603 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,605 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,605 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,608 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,609 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,609 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,709 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,711 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,712 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,713 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,713 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,714 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,714 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,714 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,714 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,715 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,715 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,715 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,715 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b2a9b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,717 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,718 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,722 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,722 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,733 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,735 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,735 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:57,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807477739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,835 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,838 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:57,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:57,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:57,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:57,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:57,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:57,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4974e271[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:57,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:57,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:57,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:57,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:57,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:57,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:57,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:57,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:57,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:57,850 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:57,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,851 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:57,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:57,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:57,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/9cfaef0ece064573af6551d2f41b34e7 2023-04-06 18:56:57,854 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,854 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:57,854 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:57,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:57,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:57,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:57,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ac3f9aba41c045099b2a12307acee040 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040 2023-04-06 18:56:57,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040, entries=200, sequenceid=356, filesize=32.3 K 2023-04-06 18:56:57,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/3579c2ea9cad4693b6ac454533a5ffbf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf 2023-04-06 18:56:57,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf, entries=150, sequenceid=356, filesize=11.9 K 2023-04-06 18:56:57,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/9cfaef0ece064573af6551d2f41b34e7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7 2023-04-06 18:56:57,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7, entries=150, sequenceid=356, filesize=11.9 K 2023-04-06 18:56:57,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for e8550365eba5972a2436e7d067fedd10 in 1273ms, sequenceid=356, compaction requested=true 2023-04-06 18:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:57,877 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:57,877 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:57,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:57,878 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:57,878 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111516 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:57,878 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:56:57,878 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:56:57,878 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,878 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,879 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad7368e13e4a43b1939a8263b374fe2c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=47.9 K 2023-04-06 18:56:57,879 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8e0c8cef5e8e4fcd94060fd8113b6565, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=108.9 K 2023-04-06 18:56:57,879 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ad7368e13e4a43b1939a8263b374fe2c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807411013 2023-04-06 18:56:57,879 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 8e0c8cef5e8e4fcd94060fd8113b6565, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807411013 2023-04-06 18:56:57,879 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 923ff00d89b64a5ca9e025ef16ba260c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680807413740 2023-04-06 18:56:57,879 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 8b43c45393f54d50afefe204c55926f5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680807413740 2023-04-06 18:56:57,879 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ad75aa486a2145b59a6a4defca31477e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680807414865 2023-04-06 18:56:57,880 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 38aaccd21bc344c6ab4d96b5658c83ee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680807414865 2023-04-06 18:56:57,880 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3579c2ea9cad4693b6ac454533a5ffbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:56:57,880 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ac3f9aba41c045099b2a12307acee040, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:56:57,905 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:57,914 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#162 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:57,931 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/6d748a8cc7594398a73ac94ad623dcd6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/6d748a8cc7594398a73ac94ad623dcd6 2023-04-06 18:56:57,938 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 6d748a8cc7594398a73ac94ad623dcd6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:57,938 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:57,938 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=12, startTime=1680807417877; duration=0sec 2023-04-06 18:56:57,938 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:57,938 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:56:57,940 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:56:57,940 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:56:57,940 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,940 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/a416bd3b3abb44fda8eb8500fd012619, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=47.9 K 2023-04-06 18:56:57,940 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a416bd3b3abb44fda8eb8500fd012619, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680807411013 2023-04-06 18:56:57,940 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 62d42d775c9f4f508c7c4073f58fef67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680807413740 2023-04-06 18:56:57,941 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 005d84cca8764ab08c73d6ce9de38a3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680807414865 2023-04-06 18:56:57,941 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9cfaef0ece064573af6551d2f41b34e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:56:57,949 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#163 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:57,950 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/8243be517294414bb644cc51f903b813 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8243be517294414bb644cc51f903b813 2023-04-06 18:56:57,954 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:57,955 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 8243be517294414bb644cc51f903b813(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:57,955 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:57,955 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=12, startTime=1680807417877; duration=0sec 2023-04-06 18:56:57,955 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:57,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:57,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:57,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:57,962 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:57,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:57,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:57,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:57,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:57,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:57,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:57,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:57,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:57,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:57,970 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:57,971 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:57,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:58,004 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304064f974d5754704cddad0aa5ddcf6eef79 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064f974d5754704cddad0aa5ddcf6eef79 2023-04-06 18:56:58,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:58,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807418133, max timestamp=1680807418157Num uniques 859; 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807418133, max timestamp=1680807418157Num uniques 859; 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807418133, max timestamp=1680807418157Num uniques 859; 2023-04-06 18:56:58,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:56:58,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807478175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807478175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807478176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807478277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807478277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807478281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,369 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/6941f4d7da664196bcaf6b46d8db79e4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6941f4d7da664196bcaf6b46d8db79e4 2023-04-06 18:56:58,376 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 6941f4d7da664196bcaf6b46d8db79e4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:56:58,376 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:58,376 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=12, startTime=1680807417877; duration=0sec 2023-04-06 18:56:58,376 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:56:58,410 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/69e2717e2b724189acc0b7b63c12403f 2023-04-06 18:56:58,430 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/a6b1daa6b9a64c2e9e3ddd72461e67c6 2023-04-06 18:56:58,447 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/28b6a3b6d00a4e5f8c5f04388bf36abf 2023-04-06 18:56:58,452 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/69e2717e2b724189acc0b7b63c12403f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f 2023-04-06 18:56:58,455 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f, entries=150, sequenceid=367, filesize=25.4 K 2023-04-06 18:56:58,457 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/a6b1daa6b9a64c2e9e3ddd72461e67c6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6 2023-04-06 18:56:58,461 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 18:56:58,462 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/28b6a3b6d00a4e5f8c5f04388bf36abf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf 2023-04-06 18:56:58,466 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 18:56:58,467 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for e8550365eba5972a2436e7d067fedd10 in 496ms, sequenceid=367, compaction requested=false 2023-04-06 18:56:58,467 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:58,467 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:58,467 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:58,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:58,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3b15b5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:58,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:58,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,472 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:58,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:58,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:58,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:58,473 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:58,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:58,476 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:58,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:58,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:58,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:58,476 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:58,477 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,477 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:56:58,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=147.60 KB heapSize=415.69 KB 2023-04-06 18:56:58,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:58,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807418163, max timestamp=1680807418480Num uniques 1118; 2023-04-06 18:56:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:56:58,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:58,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:58,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:58,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:56:58,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:58,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-06 18:56:58,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:56:58,483 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:58,483 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:58,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38298 deadline: 1680807478528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807478529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807478530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,583 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:58,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:58,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:58,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:58,585 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:58,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:58,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:58,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,586 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:58,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:58,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:58,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:58,588 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,588 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:58,588 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:58,588 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:58,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:58,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,589 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:58,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbf3a5a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:58,590 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:58,590 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,591 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:58,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:58,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:58,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:58,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:58,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:58,594 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:58,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:58,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:58,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:58,594 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:58,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,595 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:58,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:58,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,598 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,598 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:58,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807478634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807478634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807478634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,699 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:58,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:58,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:58,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:58,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:58,701 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:58,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:58,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:58,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:58,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:58,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:58,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:58,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:58,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:58,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:58,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:58,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:58,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:58,705 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:58,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:58,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740bb2c4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:58,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,707 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:58,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:58,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:58,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:58,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:58,708 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:58,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:58,710 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:58,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:58,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:58,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:58,710 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:58,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,713 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:58,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:58,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:58,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,714 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:58,714 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:58,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807478744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,814 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:58,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:58,817 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:58,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:58,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:58,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:58,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:58,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:58,821 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,821 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:58,821 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:58,821 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:58,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:58,822 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,822 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,822 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,822 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:58,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a82b15e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:58,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:58,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:58,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:58,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:58,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:58,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:58,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:58,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:58,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:58,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:58,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:58,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:58,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:58,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,830 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:58,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:58,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,834 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:58,834 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807478837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807478837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38276 deadline: 1680807478837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,898 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040665768ca73c4343bf9e81c95022d058ac to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040665768ca73c4343bf9e81c95022d058ac 2023-04-06 18:56:58,934 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:58,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:58,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:58,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:58,937 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:58,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:58,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:58,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:58,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:58,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:58,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:58,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:58,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:58,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:58,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:58,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:58,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:58,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:58,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:58,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:58,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:58,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e208906[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:58,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:58,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:58,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:58,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:58,954 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:58,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:58,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:58,956 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:58,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:58,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:58,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:58,956 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:58,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:58,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:58,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:58,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:58,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:58,960 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:58,960 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:58,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:58,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:58,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,060 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,063 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,063 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,064 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,064 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,064 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,064 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,066 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781b1b72[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,071 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,071 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,076 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,078 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,078 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807479141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807479142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807479142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,178 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,182 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,185 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,185 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,185 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,185 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,186 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51af0b27[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,188 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,189 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,191 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,191 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,197 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,197 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,197 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,297 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,300 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,301 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,301 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,301 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,301 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,302 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,302 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,302 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,302 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,302 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,303 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,303 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,305 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=406, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/e8a537893d8a498fba8aa216acde5661 2023-04-06 18:56:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e28580d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,306 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,307 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,310 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,310 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,311 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,311 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,311 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,313 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,314 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,315 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:59,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,316 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,316 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/bf9bb68f1fc24f24b542b428d8905e4a 2023-04-06 18:56:59,416 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,420 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,423 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,423 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,423 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,423 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,423 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,424 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,424 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,424 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,424 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ea8277[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,426 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,426 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,426 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,427 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,429 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,429 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,432 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,433 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,433 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,433 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,433 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:59,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,533 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,535 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,537 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,538 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@540ad713[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,541 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,541 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,541 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,541 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,542 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,544 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,544 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,545 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,545 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,545 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,545 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,549 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,549 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,649 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807479649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,652 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807479650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:56:59,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807479653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,655 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,655 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,655 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:56:59,655 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:56:59,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:56:59,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:56:59,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:56:59,656 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,656 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,656 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:56:59,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:56:59,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d550886[Count = 0] remaining members to acquire global barrier 2023-04-06 18:56:59,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,658 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:56:59,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:56:59,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:56:59,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:56:59,659 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,659 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,659 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,659 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:56:59,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:56:59,660 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:56:59,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:56:59,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:56:59,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:56:59,660 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:56:59,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:56:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:56:59,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:56:59,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,662 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:56:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:56:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,664 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:56:59,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:56:59,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:56:59,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:56:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:56:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:56:59,665 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:56:59,665 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:56:59,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/af52c9dae5fb438faee34c78a23ed879 2023-04-06 18:56:59,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/e8a537893d8a498fba8aa216acde5661 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661 2023-04-06 18:56:59,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661, entries=250, sequenceid=406, filesize=39.2 K 2023-04-06 18:56:59,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/bf9bb68f1fc24f24b542b428d8905e4a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a 2023-04-06 18:56:59,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a, entries=150, sequenceid=406, filesize=11.9 K 2023-04-06 18:56:59,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/af52c9dae5fb438faee34c78a23ed879 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879 2023-04-06 18:56:59,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879, entries=150, sequenceid=406, filesize=11.9 K 2023-04-06 18:56:59,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~317.55 KB/325168, currentSize=67.09 KB/68700 for e8550365eba5972a2436e7d067fedd10 in 1275ms, sequenceid=406, compaction requested=true 2023-04-06 18:56:59,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:56:59,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:59,756 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:59,756 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:56:59,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:59,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:56:59,757 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:59,757 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:56:59,758 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,758 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/6d748a8cc7594398a73ac94ad623dcd6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.2 K 2023-04-06 18:56:59,758 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6d748a8cc7594398a73ac94ad623dcd6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:56:59,758 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92700 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:56:59,758 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:56:59,758 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,758 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8243be517294414bb644cc51f903b813, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=90.5 K 2023-04-06 18:56:59,758 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a6b1daa6b9a64c2e9e3ddd72461e67c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807416610 2023-04-06 18:56:59,759 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8243be517294414bb644cc51f903b813, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:56:59,759 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting bf9bb68f1fc24f24b542b428d8905e4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418175 2023-04-06 18:56:59,759 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 69e2717e2b724189acc0b7b63c12403f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807416610 2023-04-06 18:56:59,759 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e8a537893d8a498fba8aa216acde5661, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418163 2023-04-06 18:56:59,764 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:59,765 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:56:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:56:59,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:56:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:56:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:56:59,769 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:56:59,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:56:59,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:56:59,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,770 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#171 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:56:59,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:56:59,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:56:59,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:56:59,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:56:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:56:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:56:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:56:59,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:56:59,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:56:59,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:56:59,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:56:59,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:56:59,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:56:59,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:56:59,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:56:59,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:56:59,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:56:59,791 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:56:59,791 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:00,186 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/3e46af3ac4b94a60bc023536702d3e16 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3e46af3ac4b94a60bc023536702d3e16 2023-04-06 18:57:00,191 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 3e46af3ac4b94a60bc023536702d3e16(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:00,191 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:00,191 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=13, startTime=1680807419756; duration=0sec 2023-04-06 18:57:00,191 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:00,191 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:00,192 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:00,192 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:57:00,192 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,192 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6941f4d7da664196bcaf6b46d8db79e4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.2 K 2023-04-06 18:57:00,193 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6941f4d7da664196bcaf6b46d8db79e4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1680807416010 2023-04-06 18:57:00,193 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 28b6a3b6d00a4e5f8c5f04388bf36abf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807416610 2023-04-06 18:57:00,194 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting af52c9dae5fb438faee34c78a23ed879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418175 2023-04-06 18:57:00,198 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:00,213 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/35923429e22a4dc397b769fa286af05e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/35923429e22a4dc397b769fa286af05e 2023-04-06 18:57:00,216 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a3b1a3d505514f78b7593fc0a67b8a0d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a3b1a3d505514f78b7593fc0a67b8a0d 2023-04-06 18:57:00,219 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ca599be9a6a648da90533f3ee77789f8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ca599be9a6a648da90533f3ee77789f8 2023-04-06 18:57:00,221 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 35923429e22a4dc397b769fa286af05e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:00,221 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:00,221 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=13, startTime=1680807419756; duration=0sec 2023-04-06 18:57:00,221 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:00,225 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into ca599be9a6a648da90533f3ee77789f8(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:00,225 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:00,225 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=13, startTime=1680807419756; duration=0sec 2023-04-06 18:57:00,225 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:00,232 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/7ba48d2a67dd4ad8b008417d7f3d87d6 2023-04-06 18:57:00,242 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/8189e4d0400447cea006b4b3cf11a9b8 2023-04-06 18:57:00,657 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/2655652119e84b37a80bc5c6b1c3176b 2023-04-06 18:57:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:00,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:00,661 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/7ba48d2a67dd4ad8b008417d7f3d87d6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6 2023-04-06 18:57:00,665 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6, entries=150, sequenceid=419, filesize=25.4 K 2023-04-06 18:57:00,666 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/8189e4d0400447cea006b4b3cf11a9b8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8 2023-04-06 18:57:00,670 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8, entries=150, sequenceid=419, filesize=11.9 K 2023-04-06 18:57:00,671 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/2655652119e84b37a80bc5c6b1c3176b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b 2023-04-06 18:57:00,674 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b, entries=150, sequenceid=419, filesize=11.9 K 2023-04-06 18:57:00,675 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=87.22 KB/89310 for e8550365eba5972a2436e7d067fedd10 in 884ms, sequenceid=419, compaction requested=false 2023-04-06 18:57:00,675 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:00,675 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:00,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:00,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:00,676 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:00,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a1df11[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:00,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:00,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,679 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:00,679 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:00,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:00,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:00,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:00,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:00,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:00,685 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:00,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:00,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:00,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:00,686 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:00,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807480692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807480695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807480696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,701 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:00,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:00,701 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807480751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,753 DEBUG [Thread-933] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807480797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807480797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:00,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807480799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,801 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:00,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:00,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:00,804 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:00,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:00,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:00,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:00,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,805 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:00,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:00,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:00,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:00,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:00,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:00,807 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f81068[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:00,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:00,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:00,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:00,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:00,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,810 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:00,810 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:00,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:00,812 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:00,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:00,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:00,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:00,812 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:00,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:00,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:00,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,816 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:00,816 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:00,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:00,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,916 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:00,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:00,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:00,919 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:00,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:00,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:00,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:00,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:00,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:00,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:00,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:00,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:00,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:00,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:00,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:00,923 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,923 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,923 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:00,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:00,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c8f50b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:00,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:00,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:00,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:00,926 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,926 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:00,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:00,928 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:00,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:00,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:00,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:00,928 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:00,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:00,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:00,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:00,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:00,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:00,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:00,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:00,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,932 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:00,932 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:00,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:00,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:00,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:00,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:00,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:00,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807481000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807480998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807481001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,034 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,036 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,039 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,039 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,039 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,039 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:01,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:01,040 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,040 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,040 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:01,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:01,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160cd4cc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:01,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,042 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:01,042 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:01,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:01,044 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:01,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:01,046 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:01,046 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:01,046 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:01,046 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:01,046 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:01,047 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,047 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:01,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,050 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:01,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,050 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,093 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406241bc3d941c44e26a3e7a9afbfc3b99d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406241bc3d941c44e26a3e7a9afbfc3b99d 2023-04-06 18:57:01,150 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,153 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,154 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,154 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,154 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:01,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,155 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,155 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,155 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,155 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:01,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:01,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,156 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:01,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:01,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@793ac84b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:01,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:01,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:01,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:01,158 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,158 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:01,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,159 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,159 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,159 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:01,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:01,160 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:01,161 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:01,161 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:01,161 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:01,161 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:01,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:01,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,164 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:01,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,164 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,265 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,267 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:01,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:01,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:01,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7840fc54[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:01,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:01,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:01,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:01,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:01,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:01,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:01,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:01,276 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:01,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:01,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:01,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:01,276 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:01,276 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,276 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,276 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,277 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,277 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:01,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,279 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,279 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:01,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807481303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807481303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807481303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38338 deadline: 1680807481303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,307 DEBUG [Thread-935] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:01,379 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,383 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,383 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:01,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,385 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,385 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,385 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,385 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:01,386 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:01,387 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:01,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c242efa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:01,387 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:01,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:01,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:01,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:01,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:01,391 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:01,391 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:01,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:01,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:01,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,395 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:01,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,396 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:01,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,496 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,497 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/21720f0e0f7a42a983a911fa4bbe69f4 2023-04-06 18:57:01,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,499 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,500 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:01,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:01,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:01,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,502 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:01,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7e2f06[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:01,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:01,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:01,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:01,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:01,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:01,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:01,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/2c7224a8a97f4490ad9f467d95d541ad 2023-04-06 18:57:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:01,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:01,509 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:01,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:01,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:01,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:01,509 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:01,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:01,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:01,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:01,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:01,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:01,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:01,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:01,514 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:01,514 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:01,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:01,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:01,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/ab0ab584961749e89fb8732b11b135eb 2023-04-06 18:57:01,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/21720f0e0f7a42a983a911fa4bbe69f4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4 2023-04-06 18:57:01,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4, entries=500, sequenceid=439, filesize=73.6 K 2023-04-06 18:57:01,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/2c7224a8a97f4490ad9f467d95d541ad as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad 2023-04-06 18:57:01,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad, entries=150, sequenceid=439, filesize=11.9 K 2023-04-06 18:57:01,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/ab0ab584961749e89fb8732b11b135eb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb 2023-04-06 18:57:01,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb, entries=150, sequenceid=439, filesize=11.9 K 2023-04-06 18:57:01,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e8550365eba5972a2436e7d067fedd10 in 866ms, sequenceid=439, compaction requested=true 2023-04-06 18:57:01,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:01,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:01,542 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:01,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:01,542 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:01,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:01,543 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 128088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:01,543 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:57:01,543 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,543 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ca599be9a6a648da90533f3ee77789f8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=125.1 K 2023-04-06 18:57:01,544 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:01,544 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ca599be9a6a648da90533f3ee77789f8, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418175 2023-04-06 18:57:01,544 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:57:01,544 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,544 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3e46af3ac4b94a60bc023536702d3e16, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.3 K 2023-04-06 18:57:01,544 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3e46af3ac4b94a60bc023536702d3e16, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418175 2023-04-06 18:57:01,544 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7ba48d2a67dd4ad8b008417d7f3d87d6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680807418494 2023-04-06 18:57:01,545 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8189e4d0400447cea006b4b3cf11a9b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680807418494 2023-04-06 18:57:01,545 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 21720f0e0f7a42a983a911fa4bbe69f4, keycount=500, bloomtype=ROW, size=73.6 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420661 2023-04-06 18:57:01,545 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2c7224a8a97f4490ad9f467d95d541ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420669 2023-04-06 18:57:01,551 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#179 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:01,589 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#180 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:01,614 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:01,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:01,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:01,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:01,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:01,617 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:01,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:01,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:01,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:01,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:01,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:01,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:01,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:01,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:01,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:01,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:01,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:01,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:01,620 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:01,620 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:01,634 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040672573f8bbe304b7e9a8c6154b6a7c29f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672573f8bbe304b7e9a8c6154b6a7c29f 2023-04-06 18:57:01,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:01,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:01,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807481821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807481824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807481825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807481928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807481930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807481930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:01,988 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/8dcd6e559dfa46e3a655b733adbab0d1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8dcd6e559dfa46e3a655b733adbab0d1 2023-04-06 18:57:01,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 8dcd6e559dfa46e3a655b733adbab0d1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:01,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:01,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=13, startTime=1680807421542; duration=0sec 2023-04-06 18:57:01,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:01,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:01,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:01,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:57:01,994 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:01,994 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/35923429e22a4dc397b769fa286af05e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.3 K 2023-04-06 18:57:01,995 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 35923429e22a4dc397b769fa286af05e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680807418175 2023-04-06 18:57:01,995 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2655652119e84b37a80bc5c6b1c3176b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680807418494 2023-04-06 18:57:01,997 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ab0ab584961749e89fb8732b11b135eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420669 2023-04-06 18:57:02,003 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:02,024 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/05ee8a19fb9c4e42afdafdffb8c09b8e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/05ee8a19fb9c4e42afdafdffb8c09b8e 2023-04-06 18:57:02,025 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/0edc4866054a446dbd46a74eb86f6730 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/0edc4866054a446dbd46a74eb86f6730 2023-04-06 18:57:02,035 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 0edc4866054a446dbd46a74eb86f6730(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:02,035 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:02,035 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=13, startTime=1680807421542; duration=0sec 2023-04-06 18:57:02,035 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:02,036 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 05ee8a19fb9c4e42afdafdffb8c09b8e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:02,036 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:02,036 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=13, startTime=1680807421541; duration=0sec 2023-04-06 18:57:02,036 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:02,048 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ccb13552bf114f31b8580beb6ca22c1e 2023-04-06 18:57:02,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807482133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807482134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807482135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807482437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807482437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807482438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,460 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1ca4e00b0030478185e18464c2eb2060 2023-04-06 18:57:02,869 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/9e20f39528bf419ebb43b9a2d8e8876a 2023-04-06 18:57:02,873 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ccb13552bf114f31b8580beb6ca22c1e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e 2023-04-06 18:57:02,876 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e, entries=150, sequenceid=454, filesize=25.4 K 2023-04-06 18:57:02,877 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/1ca4e00b0030478185e18464c2eb2060 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060 2023-04-06 18:57:02,880 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060, entries=150, sequenceid=454, filesize=11.9 K 2023-04-06 18:57:02,881 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/9e20f39528bf419ebb43b9a2d8e8876a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a 2023-04-06 18:57:02,885 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a, entries=150, sequenceid=454, filesize=11.9 K 2023-04-06 18:57:02,886 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e8550365eba5972a2436e7d067fedd10 in 1266ms, sequenceid=454, compaction requested=false 2023-04-06 18:57:02,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:02,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:02,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:02,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:02,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:02,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:02,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:02,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2329c652[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:02,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:02,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:02,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:02,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:02,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:02,891 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:02,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,891 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:02,891 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:02,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:02,893 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:02,893 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:02,894 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,894 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,894 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:02,895 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,896 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,897 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,897 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:02,898 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:02,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:02,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:02,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:02,898 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:02,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:02,898 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:02,898 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:02,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:02,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:57:02,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:02,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:02,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:02,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:02,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:02,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807482953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807482955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807482956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:02,998 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:02,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,001 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,001 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,002 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,002 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,002 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,002 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,003 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,003 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,004 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,004 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,004 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,004 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c743c3f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,006 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,010 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,010 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,010 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,010 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,010 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,010 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,012 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,014 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,015 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,015 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807483059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807483059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807483059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,115 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,117 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,118 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,120 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa8ab5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,134 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,138 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,138 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,138 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,138 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,138 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,148 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,148 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,148 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,152 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,152 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:03,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,252 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,255 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,257 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,257 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,257 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,257 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,258 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c019393[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,261 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807483260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807483261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807483262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,263 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,263 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,274 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,274 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,274 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,360 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406be0114010e534ad5be4870dc405d546a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406be0114010e534ad5be4870dc405d546a 2023-04-06 18:57:03,374 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,377 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,382 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,382 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,382 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,382 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,383 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1177c117[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,386 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,388 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,388 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:03,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,392 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,392 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,492 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,495 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9f82c2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,503 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,503 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,504 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,504 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,504 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,506 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:03,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,508 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,508 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807483564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807483564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:03,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807483564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,608 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,611 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,611 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,612 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,613 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,613 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,613 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,613 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,613 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,614 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,614 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,614 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e512c6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,618 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,618 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,624 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,624 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,724 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,727 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,729 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,730 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,730 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49888362[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,733 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,733 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,733 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,734 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,734 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,737 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,739 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,739 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,766 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=477, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/82d0a064684d4547a5ed97291251bbfc 2023-04-06 18:57:03,839 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,841 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,844 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,844 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,844 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,844 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,845 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,845 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d89c3d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,853 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,853 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,853 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,953 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:03,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:03,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:03,956 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:03,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:03,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:03,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:03,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:03,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:03,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:03,958 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,958 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:03,958 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:03,958 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:03,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:03,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:03,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:03,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76419bb6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:03,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:03,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:03,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:03,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:03,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:03,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:03,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,962 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:03,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:03,963 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:03,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:03,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:03,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:03,963 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:03,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:03,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:03,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:03,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,967 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:03,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:03,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:03,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:03,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:03,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:03,967 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:03,967 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:03,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:04,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807484065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:04,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807484065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,067 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:04,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807484067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,070 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,071 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:04,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:04,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:04,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:04,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:04,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6723d49a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:04,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:04,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:04,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:04,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:04,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:04,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:04,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:04,080 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:04,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:04,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:04,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:04,080 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:04,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:04,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:04,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:04,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,084 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,084 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:04,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/e1508a9683164257a8b1d96a3963fa25 2023-04-06 18:57:04,184 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,187 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,188 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:04,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,190 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,190 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,190 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:04,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:04,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:04,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3dc88f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:04,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:04,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:04,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:04,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:04,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:04,193 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:04,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:04,211 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:04,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:04,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:04,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:04,211 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:04,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:04,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:04,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,216 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:04,216 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:04,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,316 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,318 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,321 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,321 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,321 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,321 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:04,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:04,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,322 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:04,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48268881[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:04,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:04,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,323 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:04,323 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:04,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:04,323 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:04,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:04,324 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:04,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:04,326 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:04,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:04,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:04,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:04,326 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:04,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:04,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,329 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:04,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:04,329 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:04,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,330 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,330 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,430 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,433 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,433 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,434 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:04,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,435 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,435 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,435 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,435 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:04,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:04,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:04,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:04,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32403618[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:04,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:04,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:04,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:04,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:04,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:04,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:04,441 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:04,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:04,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:04,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:04,441 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:04,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,443 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:04,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:04,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,445 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:04,445 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:04,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:04,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,545 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,548 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,549 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:04,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,550 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:04,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:04,551 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:04,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3ea3ce[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:04,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:04,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:04,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:04,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:04,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:04,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:04,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:04,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:04,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:04,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:04,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:04,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:04,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,558 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,559 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:04,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:04,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:04,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,560 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:04,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:04,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:04,560 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:04,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:04,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/dc560067d24c49729e18737c75a2e2e1 2023-04-06 18:57:04,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/82d0a064684d4547a5ed97291251bbfc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc 2023-04-06 18:57:04,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc, entries=200, sequenceid=477, filesize=32.3 K 2023-04-06 18:57:04,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/e1508a9683164257a8b1d96a3963fa25 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25 2023-04-06 18:57:04,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25, entries=150, sequenceid=477, filesize=11.9 K 2023-04-06 18:57:04,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/dc560067d24c49729e18737c75a2e2e1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1 2023-04-06 18:57:04,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1, entries=150, sequenceid=477, filesize=11.9 K 2023-04-06 18:57:04,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e8550365eba5972a2436e7d067fedd10 in 1659ms, sequenceid=477, compaction requested=true 2023-04-06 18:57:04,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:04,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:04,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:04,604 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:04,604 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:04,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:04,605 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:04,605 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:57:04,605 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:04,605 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,605 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:57:04,605 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8dcd6e559dfa46e3a655b733adbab0d1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.4 K 2023-04-06 18:57:04,605 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,606 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/05ee8a19fb9c4e42afdafdffb8c09b8e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=83.8 K 2023-04-06 18:57:04,606 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8dcd6e559dfa46e3a655b733adbab0d1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420669 2023-04-06 18:57:04,606 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 05ee8a19fb9c4e42afdafdffb8c09b8e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420669 2023-04-06 18:57:04,606 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1ca4e00b0030478185e18464c2eb2060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680807420691 2023-04-06 18:57:04,606 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ccb13552bf114f31b8580beb6ca22c1e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680807420691 2023-04-06 18:57:04,607 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e1508a9683164257a8b1d96a3963fa25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:04,607 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 82d0a064684d4547a5ed97291251bbfc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:04,616 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:04,618 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#188 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:04,660 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:04,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:04,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:04,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:04,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:04,663 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:04,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:04,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:04,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,664 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:04,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:04,664 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,664 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:04,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:04,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:04,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:04,665 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:04,665 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:04,665 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:04,665 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:04,665 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:04,665 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:04,665 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:04,665 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:57:04,665 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:04,665 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:04,665 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:04,666 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:04,666 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:04,666 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:04,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:04,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:04,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:04,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807484827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:04,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:04,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807484930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,026 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/838572fa316646748a93ba1ad830b384 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/838572fa316646748a93ba1ad830b384 2023-04-06 18:57:05,031 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 838572fa316646748a93ba1ad830b384(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:05,031 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:05,031 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=13, startTime=1680807424604; duration=0sec 2023-04-06 18:57:05,031 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:05,031 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:05,033 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:05,033 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:57:05,033 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:05,033 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/0edc4866054a446dbd46a74eb86f6730, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=36.4 K 2023-04-06 18:57:05,033 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0edc4866054a446dbd46a74eb86f6730, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680807420669 2023-04-06 18:57:05,034 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9e20f39528bf419ebb43b9a2d8e8876a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680807420691 2023-04-06 18:57:05,034 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting dc560067d24c49729e18737c75a2e2e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:05,037 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/c7bd4ae63c634210a4295e5811179af9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/c7bd4ae63c634210a4295e5811179af9 2023-04-06 18:57:05,043 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#191 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:05,047 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into c7bd4ae63c634210a4295e5811179af9(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:05,047 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:05,047 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=13, startTime=1680807424604; duration=0sec 2023-04-06 18:57:05,047 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:05,060 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/ece98833fffc48d98151a315033bc336 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ece98833fffc48d98151a315033bc336 2023-04-06 18:57:05,064 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into ece98833fffc48d98151a315033bc336(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:05,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:05,064 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=13, startTime=1680807424604; duration=0sec 2023-04-06 18:57:05,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:05,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38298 deadline: 1680807485068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38276 deadline: 1680807485072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38312 deadline: 1680807485074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,079 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ef50bb2ead56470ca87645a45c2a5de6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ef50bb2ead56470ca87645a45c2a5de6 2023-04-06 18:57:05,085 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=489, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/b2bfe87da43a415c8637a4536218aa57 2023-04-06 18:57:05,094 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ef1a7b4b8c614035bcfe342475b636ae 2023-04-06 18:57:05,121 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/3a171f78d97d45648d2fa5303b3df394 2023-04-06 18:57:05,125 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/b2bfe87da43a415c8637a4536218aa57 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57 2023-04-06 18:57:05,131 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57, entries=150, sequenceid=489, filesize=25.4 K 2023-04-06 18:57:05,132 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ef1a7b4b8c614035bcfe342475b636ae as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae 2023-04-06 18:57:05,137 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae, entries=150, sequenceid=489, filesize=11.9 K 2023-04-06 18:57:05,138 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/3a171f78d97d45648d2fa5303b3df394 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394 2023-04-06 18:57:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38322 deadline: 1680807485134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,143 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394, entries=150, sequenceid=489, filesize=11.9 K 2023-04-06 18:57:05,143 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e8550365eba5972a2436e7d067fedd10 in 478ms, sequenceid=489, compaction requested=false 2023-04-06 18:57:05,143 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:05,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:05,146 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:05,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:05,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc36396[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:05,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:05,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:05,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:05,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:05,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:05,153 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:05,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:05,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:05,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:05,153 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:05,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:05,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:05,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:05,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:05,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:05,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:05,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:05,165 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:05,165 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:05,165 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:05,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:05,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:05,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:05,265 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:05,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:05,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:05,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:05,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:05,268 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:05,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:05,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:05,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:05,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:05,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:05,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:05,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:05,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:05,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:05,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:05,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. started... 2023-04-06 18:57:05,271 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:05,442 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. as already flushing 2023-04-06 18:57:05,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38322 deadline: 1680807485456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38322 deadline: 1680807485560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:05,683 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406cf70c515ea614077b5e5435c70d5b688 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cf70c515ea614077b5e5435c70d5b688 2023-04-06 18:57:05,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:05,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38322 deadline: 1680807485763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:06,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38322 deadline: 1680807486065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,088 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=512, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/67918c1a3dda430390e71288e1f335c1 2023-04-06 18:57:06,497 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/9310bf2cd4e14705901baa976da5df65 2023-04-06 18:57:06,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:06,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38322 deadline: 1680807486567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,906 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/8e8b809dff7b46c6889fcf4de2d03547 2023-04-06 18:57:06,909 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/67918c1a3dda430390e71288e1f335c1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1 2023-04-06 18:57:06,912 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1, entries=150, sequenceid=512, filesize=25.4 K 2023-04-06 18:57:06,913 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/9310bf2cd4e14705901baa976da5df65 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65 2023-04-06 18:57:06,916 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65, entries=150, sequenceid=512, filesize=11.9 K 2023-04-06 18:57:06,917 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/8e8b809dff7b46c6889fcf4de2d03547 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547 2023-04-06 18:57:06,920 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547, entries=150, sequenceid=512, filesize=11.9 K 2023-04-06 18:57:06,921 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e8550365eba5972a2436e7d067fedd10 in 1650ms, sequenceid=512, compaction requested=true 2023-04-06 18:57:06,921 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:06,921 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:06,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:06,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:06,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:06,921 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:06,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:06,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:06,923 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:06,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:06,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33dd986f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:06,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:06,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,939 DEBUG [Thread-942] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7605c204 to 127.0.0.1:63695 2023-04-06 18:57:06,939 DEBUG [Thread-942] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:06,940 DEBUG [Thread-944] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16505096 to 127.0.0.1:63695 2023-04-06 18:57:06,940 DEBUG [Thread-944] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:06,941 DEBUG [Thread-940] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c411e08 to 127.0.0.1:63695 2023-04-06 18:57:06,941 DEBUG [Thread-940] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:06,941 DEBUG [Thread-946] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03e1990e to 127.0.0.1:63695 2023-04-06 18:57:06,941 DEBUG [Thread-946] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:06,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:06,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:06,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:06,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:06,971 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:06,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:06,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:06,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:06,976 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:06,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:06,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:06,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:06,976 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:06,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:06,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:06,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:06,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:06,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:06,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:06,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:06,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:06,984 DEBUG [Thread-939] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:06,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:06,984 DEBUG [Thread-939] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:06,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:07,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:07,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:57:07,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:07,079 DEBUG [Thread-929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b75e45b to 127.0.0.1:63695 2023-04-06 18:57:07,079 DEBUG [Thread-929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:07,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:07,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:07,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:07,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:07,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:07,084 DEBUG [Thread-939] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:07,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:07,087 DEBUG [Thread-937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43a14bcf to 127.0.0.1:63695 2023-04-06 18:57:07,087 DEBUG [Thread-937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:07,092 DEBUG [Thread-931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6017755a to 127.0.0.1:63695 2023-04-06 18:57:07,092 DEBUG [Thread-931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:07,504 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065b64e691b9be40849913bab10139bd28 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065b64e691b9be40849913bab10139bd28 2023-04-06 18:57:07,575 DEBUG [Thread-933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x633c81ba to 127.0.0.1:63695 2023-04-06 18:57:07,576 DEBUG [Thread-933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:07,912 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=525, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/20615d8b820b4e0681074c47751a1eca 2023-04-06 18:57:08,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/645c09e4d0b4490bafd102320663289a 2023-04-06 18:57:08,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/fad91b5ce227454ea556d55d40a70879 2023-04-06 18:57:08,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/20615d8b820b4e0681074c47751a1eca as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca 2023-04-06 18:57:08,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca, entries=150, sequenceid=525, filesize=25.4 K 2023-04-06 18:57:08,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/645c09e4d0b4490bafd102320663289a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a 2023-04-06 18:57:08,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a, entries=150, sequenceid=525, filesize=11.9 K 2023-04-06 18:57:08,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/fad91b5ce227454ea556d55d40a70879 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879 2023-04-06 18:57:08,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879, entries=150, sequenceid=525, filesize=11.9 K 2023-04-06 18:57:08,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for e8550365eba5972a2436e7d067fedd10 in 1698ms, sequenceid=525, compaction requested=true 2023-04-06 18:57:08,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:08,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:08,776 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:08,776 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:08,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:08,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/B is initiating minor compaction (all files) 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/A is initiating minor compaction (all files) 2023-04-06 18:57:08,778 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/B in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:08,778 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/A in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:08,778 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/c7bd4ae63c634210a4295e5811179af9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=102.5 K 2023-04-06 18:57:08,778 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/838572fa316646748a93ba1ad830b384, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=48.4 K 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c7bd4ae63c634210a4295e5811179af9, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 838572fa316646748a93ba1ad830b384, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b2bfe87da43a415c8637a4536218aa57, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680807422948 2023-04-06 18:57:08,778 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ef1a7b4b8c614035bcfe342475b636ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680807422948 2023-04-06 18:57:08,779 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 67918c1a3dda430390e71288e1f335c1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680807424818 2023-04-06 18:57:08,779 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9310bf2cd4e14705901baa976da5df65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680807424818 2023-04-06 18:57:08,779 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 20615d8b820b4e0681074c47751a1eca, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680807425451 2023-04-06 18:57:08,779 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 645c09e4d0b4490bafd102320663289a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680807425451 2023-04-06 18:57:08,786 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:08,788 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#A#compaction#201 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:09,202 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/232902a81d1d46cb8fad373d62340219 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/232902a81d1d46cb8fad373d62340219 2023-04-06 18:57:09,208 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/B of e8550365eba5972a2436e7d067fedd10 into 232902a81d1d46cb8fad373d62340219(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:09,208 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:09,208 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/B, priority=12, startTime=1680807428776; duration=0sec 2023-04-06 18:57:09,208 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:09,209 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:09,210 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:09,210 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): e8550365eba5972a2436e7d067fedd10/C is initiating minor compaction (all files) 2023-04-06 18:57:09,210 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e8550365eba5972a2436e7d067fedd10/C in TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:09,210 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ece98833fffc48d98151a315033bc336, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp, totalSize=48.4 K 2023-04-06 18:57:09,211 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ece98833fffc48d98151a315033bc336, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680807421818 2023-04-06 18:57:09,211 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3a171f78d97d45648d2fa5303b3df394, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680807422948 2023-04-06 18:57:09,212 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8e8b809dff7b46c6889fcf4de2d03547, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1680807424818 2023-04-06 18:57:09,212 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting fad91b5ce227454ea556d55d40a70879, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680807425451 2023-04-06 18:57:09,220 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): e8550365eba5972a2436e7d067fedd10#C#compaction#202 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:09,632 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/6d80bec10d2a452ab130722923fe4290 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6d80bec10d2a452ab130722923fe4290 2023-04-06 18:57:09,640 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/C of e8550365eba5972a2436e7d067fedd10 into 6d80bec10d2a452ab130722923fe4290(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:09,640 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:09,641 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/C, priority=12, startTime=1680807428777; duration=0sec 2023-04-06 18:57:09,641 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:10,020 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/2a06731323ab4db6bf176991e980ec4a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a06731323ab4db6bf176991e980ec4a 2023-04-06 18:57:10,024 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e8550365eba5972a2436e7d067fedd10/A of e8550365eba5972a2436e7d067fedd10 into 2a06731323ab4db6bf176991e980ec4a(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 18:57:10,024 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:10,024 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10., storeName=e8550365eba5972a2436e7d067fedd10/A, priority=12, startTime=1680807428776; duration=1sec 2023-04-06 18:57:10,024 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:11,409 DEBUG [Thread-935] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x661b21c2 to 127.0.0.1:63695 2023-04-06 18:57:11,409 DEBUG [Thread-935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:57:11,409 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 6293 2023-04-06 18:57:11,410 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 6116 2023-04-06 18:57:11,410 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:57:11,411 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2550 2023-04-06 18:57:11,411 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 7650 rows 2023-04-06 18:57:11,411 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2574 2023-04-06 18:57:11,411 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 7722 rows 2023-04-06 18:57:11,411 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:57:11,411 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f279d2 to 127.0.0.1:63695 2023-04-06 18:57:11,411 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:11,412 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:57:11,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:57:11,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:11,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:11,426 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807431425"}]},"ts":"1680807431425"} 2023-04-06 18:57:11,426 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:57:11,427 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:57:11,428 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, UNASSIGN}] 2023-04-06 18:57:11,429 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, UNASSIGN 2023-04-06 18:57:11,430 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:11,431 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807431430"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807431430"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807431430"}]},"ts":"1680807431430"} 2023-04-06 18:57:11,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:11,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:11,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing e8550365eba5972a2436e7d067fedd10, disabling compactions & flushes 2023-04-06 18:57:11,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. after waiting 0 ms 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:11,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing e8550365eba5972a2436e7d067fedd10 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=A 2023-04-06 18:57:11,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:11,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=B 2023-04-06 18:57:11,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:11,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e8550365eba5972a2436e7d067fedd10, store=C 2023-04-06 18:57:11,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:11,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040653c2c262365e4c9bbfc23d5e38a4b69f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c2c262365e4c9bbfc23d5e38a4b69f 2023-04-06 18:57:11,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:12,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=535, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ce0cfcee7c46460cb0729be04f8d2719 2023-04-06 18:57:12,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:12,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ca9b3ff29afa43d197a1ff33e43aa94e 2023-04-06 18:57:12,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:12,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/90f77485fd594efb9b41903f89e69d54 2023-04-06 18:57:12,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/A/ce0cfcee7c46460cb0729be04f8d2719 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ce0cfcee7c46460cb0729be04f8d2719 2023-04-06 18:57:12,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ce0cfcee7c46460cb0729be04f8d2719, entries=150, sequenceid=535, filesize=25.4 K 2023-04-06 18:57:12,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/B/ca9b3ff29afa43d197a1ff33e43aa94e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ca9b3ff29afa43d197a1ff33e43aa94e 2023-04-06 18:57:12,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ca9b3ff29afa43d197a1ff33e43aa94e, entries=150, sequenceid=535, filesize=11.9 K 2023-04-06 18:57:12,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/.tmp/C/90f77485fd594efb9b41903f89e69d54 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/90f77485fd594efb9b41903f89e69d54 2023-04-06 18:57:12,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/90f77485fd594efb9b41903f89e69d54, entries=150, sequenceid=535, filesize=11.9 K 2023-04-06 18:57:12,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e8550365eba5972a2436e7d067fedd10 in 1273ms, sequenceid=535, compaction requested=false 2023-04-06 18:57:12,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9570124a403d46c783fb7910974bb081, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/201a412165b7476f92fbcb78ab50e7bb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8e0c8cef5e8e4fcd94060fd8113b6565, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8243be517294414bb644cc51f903b813, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ca599be9a6a648da90533f3ee77789f8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/05ee8a19fb9c4e42afdafdffb8c09b8e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/c7bd4ae63c634210a4295e5811179af9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca] to archive 2023-04-06 18:57:12,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:12,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b736476306714bf8bb04084d237d1fb5 2023-04-06 18:57:12,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/3d05f996cdee45d8b654c6db9aa79633 2023-04-06 18:57:12,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7db4f847f98f4331996408484d1650c6 2023-04-06 18:57:12,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/af7e1a6e2d9c4aa491bc587cbb776c86 2023-04-06 18:57:12,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9570124a403d46c783fb7910974bb081 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9570124a403d46c783fb7910974bb081 2023-04-06 18:57:12,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/9a543d7f87444b7691efbedafbab69fd 2023-04-06 18:57:12,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a3f6fc016964dd7b7869e3494321907 2023-04-06 18:57:12,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/201a412165b7476f92fbcb78ab50e7bb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/201a412165b7476f92fbcb78ab50e7bb 2023-04-06 18:57:12,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/bbd0cb38e0944300a264b024b22ba16d 2023-04-06 18:57:12,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8e0c8cef5e8e4fcd94060fd8113b6565 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8e0c8cef5e8e4fcd94060fd8113b6565 2023-04-06 18:57:12,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2aa2d00366014b689db3cea19a8eb938 2023-04-06 18:57:12,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8b43c45393f54d50afefe204c55926f5 2023-04-06 18:57:12,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/38aaccd21bc344c6ab4d96b5658c83ee 2023-04-06 18:57:12,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ac3f9aba41c045099b2a12307acee040 2023-04-06 18:57:12,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8243be517294414bb644cc51f903b813 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/8243be517294414bb644cc51f903b813 2023-04-06 18:57:12,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/69e2717e2b724189acc0b7b63c12403f 2023-04-06 18:57:12,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/e8a537893d8a498fba8aa216acde5661 2023-04-06 18:57:12,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ca599be9a6a648da90533f3ee77789f8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ca599be9a6a648da90533f3ee77789f8 2023-04-06 18:57:12,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/7ba48d2a67dd4ad8b008417d7f3d87d6 2023-04-06 18:57:12,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/21720f0e0f7a42a983a911fa4bbe69f4 2023-04-06 18:57:12,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/05ee8a19fb9c4e42afdafdffb8c09b8e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/05ee8a19fb9c4e42afdafdffb8c09b8e 2023-04-06 18:57:12,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ccb13552bf114f31b8580beb6ca22c1e 2023-04-06 18:57:12,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/82d0a064684d4547a5ed97291251bbfc 2023-04-06 18:57:12,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/c7bd4ae63c634210a4295e5811179af9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/c7bd4ae63c634210a4295e5811179af9 2023-04-06 18:57:12,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/b2bfe87da43a415c8637a4536218aa57 2023-04-06 18:57:12,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/67918c1a3dda430390e71288e1f335c1 2023-04-06 18:57:12,911 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/20615d8b820b4e0681074c47751a1eca 2023-04-06 18:57:12,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/40a75bbbbe8944ed91963609ab6dac1e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ed660b8699e429d949fe68ea8375bf5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad7368e13e4a43b1939a8263b374fe2c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/6d748a8cc7594398a73ac94ad623dcd6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3e46af3ac4b94a60bc023536702d3e16, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8dcd6e559dfa46e3a655b733adbab0d1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/838572fa316646748a93ba1ad830b384, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a] to archive 2023-04-06 18:57:12,915 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:12,916 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1e6c2710b2354d059a3752d0cbecab70 2023-04-06 18:57:12,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/28850498cd04481ca067364baa76a90f 2023-04-06 18:57:12,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bed645f6ade44ae28d25e95c17494ef9 2023-04-06 18:57:12,920 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/40a75bbbbe8944ed91963609ab6dac1e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/40a75bbbbe8944ed91963609ab6dac1e 2023-04-06 18:57:12,921 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/575d5d2e74a34a6190593408cd863873 2023-04-06 18:57:12,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/cf7f5f0b15ce407b977593120689f15d 2023-04-06 18:57:12,923 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ed660b8699e429d949fe68ea8375bf5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ed660b8699e429d949fe68ea8375bf5 2023-04-06 18:57:12,924 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1d8bc5dff68c4390b98d0da809fa42e5 2023-04-06 18:57:12,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/600478d784e247a298e06eaf4700a523 2023-04-06 18:57:12,926 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad7368e13e4a43b1939a8263b374fe2c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad7368e13e4a43b1939a8263b374fe2c 2023-04-06 18:57:12,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/f6376b59f6d841df886811642cbff3eb 2023-04-06 18:57:12,928 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/923ff00d89b64a5ca9e025ef16ba260c 2023-04-06 18:57:12,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ad75aa486a2145b59a6a4defca31477e 2023-04-06 18:57:12,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/6d748a8cc7594398a73ac94ad623dcd6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/6d748a8cc7594398a73ac94ad623dcd6 2023-04-06 18:57:12,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3579c2ea9cad4693b6ac454533a5ffbf 2023-04-06 18:57:12,932 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/a6b1daa6b9a64c2e9e3ddd72461e67c6 2023-04-06 18:57:12,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3e46af3ac4b94a60bc023536702d3e16 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/3e46af3ac4b94a60bc023536702d3e16 2023-04-06 18:57:12,934 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/bf9bb68f1fc24f24b542b428d8905e4a 2023-04-06 18:57:12,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8189e4d0400447cea006b4b3cf11a9b8 2023-04-06 18:57:12,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8dcd6e559dfa46e3a655b733adbab0d1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/8dcd6e559dfa46e3a655b733adbab0d1 2023-04-06 18:57:12,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/2c7224a8a97f4490ad9f467d95d541ad 2023-04-06 18:57:12,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/1ca4e00b0030478185e18464c2eb2060 2023-04-06 18:57:12,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/838572fa316646748a93ba1ad830b384 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/838572fa316646748a93ba1ad830b384 2023-04-06 18:57:12,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/e1508a9683164257a8b1d96a3963fa25 2023-04-06 18:57:12,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ef1a7b4b8c614035bcfe342475b636ae 2023-04-06 18:57:12,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/9310bf2cd4e14705901baa976da5df65 2023-04-06 18:57:12,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/645c09e4d0b4490bafd102320663289a 2023-04-06 18:57:12,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2bc71ae6a602430c88f697213d278761, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/4a4a03a7478b415387918bdd9c63b5b9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/a416bd3b3abb44fda8eb8500fd012619, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6941f4d7da664196bcaf6b46d8db79e4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/35923429e22a4dc397b769fa286af05e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/0edc4866054a446dbd46a74eb86f6730, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ece98833fffc48d98151a315033bc336, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879] to archive 2023-04-06 18:57:12,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:12,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/67e75ba49cb24a6dbece697832af6306 2023-04-06 18:57:12,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3189ae23ccb84d949d829557bb472e7f 2023-04-06 18:57:12,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/30498960341441dabd441007b3b699dc 2023-04-06 18:57:12,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2bc71ae6a602430c88f697213d278761 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2bc71ae6a602430c88f697213d278761 2023-04-06 18:57:12,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/94e00307b3824c9d927a8bd86173e756 2023-04-06 18:57:12,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/72aec155eaf54ba1b3f523dcc223f9ed 2023-04-06 18:57:12,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/4a4a03a7478b415387918bdd9c63b5b9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/4a4a03a7478b415387918bdd9c63b5b9 2023-04-06 18:57:12,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/03d5f53593d245d7af2178e5529d6f48 2023-04-06 18:57:12,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/b7f5c6d907234729b5a55ebacb5b316c 2023-04-06 18:57:12,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/a416bd3b3abb44fda8eb8500fd012619 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/a416bd3b3abb44fda8eb8500fd012619 2023-04-06 18:57:12,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/e8d219148dcc481095345f954d566822 2023-04-06 18:57:12,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/62d42d775c9f4f508c7c4073f58fef67 2023-04-06 18:57:12,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/005d84cca8764ab08c73d6ce9de38a3c 2023-04-06 18:57:12,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6941f4d7da664196bcaf6b46d8db79e4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6941f4d7da664196bcaf6b46d8db79e4 2023-04-06 18:57:12,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9cfaef0ece064573af6551d2f41b34e7 2023-04-06 18:57:12,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/28b6a3b6d00a4e5f8c5f04388bf36abf 2023-04-06 18:57:12,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/35923429e22a4dc397b769fa286af05e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/35923429e22a4dc397b769fa286af05e 2023-04-06 18:57:12,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/af52c9dae5fb438faee34c78a23ed879 2023-04-06 18:57:12,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/2655652119e84b37a80bc5c6b1c3176b 2023-04-06 18:57:12,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/0edc4866054a446dbd46a74eb86f6730 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/0edc4866054a446dbd46a74eb86f6730 2023-04-06 18:57:12,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ab0ab584961749e89fb8732b11b135eb 2023-04-06 18:57:12,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/9e20f39528bf419ebb43b9a2d8e8876a 2023-04-06 18:57:12,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ece98833fffc48d98151a315033bc336 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/ece98833fffc48d98151a315033bc336 2023-04-06 18:57:12,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/dc560067d24c49729e18737c75a2e2e1 2023-04-06 18:57:12,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/3a171f78d97d45648d2fa5303b3df394 2023-04-06 18:57:12,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/8e8b809dff7b46c6889fcf4de2d03547 2023-04-06 18:57:12,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/fad91b5ce227454ea556d55d40a70879 2023-04-06 18:57:12,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits/538.seqid, newMaxSeqId=538, maxSeqId=4 2023-04-06 18:57:12,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10. 2023-04-06 18:57:12,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for e8550365eba5972a2436e7d067fedd10: 2023-04-06 18:57:12,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:12,979 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e8550365eba5972a2436e7d067fedd10, regionState=CLOSED 2023-04-06 18:57:12,979 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807432979"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807432979"}]},"ts":"1680807432979"} 2023-04-06 18:57:12,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-06 18:57:12,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure e8550365eba5972a2436e7d067fedd10, server=jenkins-hbase20.apache.org,34607,1680807373848 in 1.5480 sec 2023-04-06 18:57:12,983 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-06 18:57:12,983 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e8550365eba5972a2436e7d067fedd10, UNASSIGN in 1.5540 sec 2023-04-06 18:57:12,984 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807432984"}]},"ts":"1680807432984"} 2023-04-06 18:57:12,986 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:57:12,987 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:57:12,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5750 sec 2023-04-06 18:57:13,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 18:57:13,532 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-06 18:57:13,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:57:13,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,539 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 18:57:13,540 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,547 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:13,550 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits] 2023-04-06 18:57:13,554 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a06731323ab4db6bf176991e980ec4a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/2a06731323ab4db6bf176991e980ec4a 2023-04-06 18:57:13,556 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ce0cfcee7c46460cb0729be04f8d2719 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/A/ce0cfcee7c46460cb0729be04f8d2719 2023-04-06 18:57:13,559 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/232902a81d1d46cb8fad373d62340219 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/232902a81d1d46cb8fad373d62340219 2023-04-06 18:57:13,560 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ca9b3ff29afa43d197a1ff33e43aa94e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/B/ca9b3ff29afa43d197a1ff33e43aa94e 2023-04-06 18:57:13,564 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6d80bec10d2a452ab130722923fe4290 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/6d80bec10d2a452ab130722923fe4290 2023-04-06 18:57:13,566 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/90f77485fd594efb9b41903f89e69d54 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/C/90f77485fd594efb9b41903f89e69d54 2023-04-06 18:57:13,569 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits/538.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10/recovered.edits/538.seqid 2023-04-06 18:57:13,570 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/e8550365eba5972a2436e7d067fedd10 2023-04-06 18:57:13,570 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:57:13,570 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:57:13,571 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 18:57:13,576 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406241bc3d941c44e26a3e7a9afbfc3b99d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406241bc3d941c44e26a3e7a9afbfc3b99d 2023-04-06 18:57:13,577 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062c670cc3b96b413d8f7cde2ad9d23a94 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062c670cc3b96b413d8f7cde2ad9d23a94 2023-04-06 18:57:13,579 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040641c0335ea1f7459dbed35b5143a4bc9a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040641c0335ea1f7459dbed35b5143a4bc9a 2023-04-06 18:57:13,580 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064f974d5754704cddad0aa5ddcf6eef79 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064f974d5754704cddad0aa5ddcf6eef79 2023-04-06 18:57:13,581 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c2c262365e4c9bbfc23d5e38a4b69f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040653c2c262365e4c9bbfc23d5e38a4b69f 2023-04-06 18:57:13,583 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040654bea46f354446c2870bff1e6c2d059a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040654bea46f354446c2870bff1e6c2d059a 2023-04-06 18:57:13,584 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065b64e691b9be40849913bab10139bd28 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065b64e691b9be40849913bab10139bd28 2023-04-06 18:57:13,586 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040665768ca73c4343bf9e81c95022d058ac to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040665768ca73c4343bf9e81c95022d058ac 2023-04-06 18:57:13,588 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066b12393118174ab4a7ea60f4f6e97e07 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066b12393118174ab4a7ea60f4f6e97e07 2023-04-06 18:57:13,589 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672573f8bbe304b7e9a8c6154b6a7c29f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672573f8bbe304b7e9a8c6154b6a7c29f 2023-04-06 18:57:13,590 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672c1fe5a23b2486e9e4df1c2e0460ff0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040672c1fe5a23b2486e9e4df1c2e0460ff0 2023-04-06 18:57:13,592 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e09f06b525a4d8dbf8e4d49646ab8d1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e09f06b525a4d8dbf8e4d49646ab8d1 2023-04-06 18:57:13,593 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068e304ebae24f496face49d6c4175060b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068e304ebae24f496face49d6c4175060b 2023-04-06 18:57:13,594 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069434aacd2b374b13b952322121c2cc59 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069434aacd2b374b13b952322121c2cc59 2023-04-06 18:57:13,595 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a3b1a3d505514f78b7593fc0a67b8a0d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a3b1a3d505514f78b7593fc0a67b8a0d 2023-04-06 18:57:13,596 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406be0114010e534ad5be4870dc405d546a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406be0114010e534ad5be4870dc405d546a 2023-04-06 18:57:13,597 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c31750ea44a14aff8dc9882415ce1bb4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c31750ea44a14aff8dc9882415ce1bb4 2023-04-06 18:57:13,598 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cf70c515ea614077b5e5435c70d5b688 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cf70c515ea614077b5e5435c70d5b688 2023-04-06 18:57:13,599 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d117096a4165417eaeb9cfa5609e08d0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d117096a4165417eaeb9cfa5609e08d0 2023-04-06 18:57:13,600 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ec45e7aa01b94482a62a9474acc380b7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ec45e7aa01b94482a62a9474acc380b7 2023-04-06 18:57:13,600 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ef50bb2ead56470ca87645a45c2a5de6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ef50bb2ead56470ca87645a45c2a5de6 2023-04-06 18:57:13,601 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:57:13,602 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,604 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:57:13,606 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:57:13,607 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,607 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:57:13,607 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807433607"}]},"ts":"9223372036854775807"} 2023-04-06 18:57:13,609 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:57:13,609 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e8550365eba5972a2436e7d067fedd10, NAME => 'TestAcidGuarantees,,1680807403745.e8550365eba5972a2436e7d067fedd10.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:57:13,610 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:57:13,610 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807433610"}]},"ts":"9223372036854775807"} 2023-04-06 18:57:13,611 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:57:13,613 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,614 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 79 msec 2023-04-06 18:57:13,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 18:57:13,642 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-06 18:57:13,657 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232 (was 233), OpenFileDescriptor=548 (was 590), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=400 (was 421), ProcessCount=166 (was 166), AvailableMemoryMB=2912 (was 3022) 2023-04-06 18:57:13,671 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=231, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=166, AvailableMemoryMB=2912 2023-04-06 18:57:13,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:57:13,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:57:13,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:13,677 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:57:13,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-06 18:57:13,678 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:57:13,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:13,679 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:13,679 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd empty. 2023-04-06 18:57:13,680 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:13,680 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:57:13,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:13,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:14,090 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:57:14,097 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => dd972e696d2dd9f5c10736fe54de15bd, NAME => 'TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:57:14,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:14,504 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:57:14,504 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing dd972e696d2dd9f5c10736fe54de15bd, disabling compactions & flushes 2023-04-06 18:57:14,504 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,504 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,504 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. after waiting 0 ms 2023-04-06 18:57:14,505 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,505 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,505 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:14,513 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:57:14,515 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807434514"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807434514"}]},"ts":"1680807434514"} 2023-04-06 18:57:14,516 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:57:14,518 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:57:14,518 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807434518"}]},"ts":"1680807434518"} 2023-04-06 18:57:14,519 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:57:14,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, ASSIGN}] 2023-04-06 18:57:14,525 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, ASSIGN 2023-04-06 18:57:14,526 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:57:14,679 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=dd972e696d2dd9f5c10736fe54de15bd, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:14,679 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807434679"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807434679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807434679"}]},"ts":"1680807434679"} 2023-04-06 18:57:14,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:14,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:14,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => dd972e696d2dd9f5c10736fe54de15bd, NAME => 'TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:57:14,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:57:14,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,848 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,852 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A 2023-04-06 18:57:14,852 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A 2023-04-06 18:57:14,852 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:14,853 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dd972e696d2dd9f5c10736fe54de15bd columnFamilyName A 2023-04-06 18:57:14,854 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(310): Store=dd972e696d2dd9f5c10736fe54de15bd/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:14,854 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,856 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B 2023-04-06 18:57:14,856 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B 2023-04-06 18:57:14,856 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:14,856 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dd972e696d2dd9f5c10736fe54de15bd columnFamilyName B 2023-04-06 18:57:14,857 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(310): Store=dd972e696d2dd9f5c10736fe54de15bd/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:14,857 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,858 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C 2023-04-06 18:57:14,858 DEBUG [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C 2023-04-06 18:57:14,859 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:14,859 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dd972e696d2dd9f5c10736fe54de15bd columnFamilyName C 2023-04-06 18:57:14,859 INFO [StoreOpener-dd972e696d2dd9f5c10736fe54de15bd-1] regionserver.HStore(310): Store=dd972e696d2dd9f5c10736fe54de15bd/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:14,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:57:14,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:14,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:57:14,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened dd972e696d2dd9f5c10736fe54de15bd; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63664659, jitterRate=-0.05132265388965607}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:57:14,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:14,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., pid=30, masterSystemTime=1680807434836 2023-04-06 18:57:14,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:14,871 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=dd972e696d2dd9f5c10736fe54de15bd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:14,871 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807434871"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807434871"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807434871"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807434871"}]},"ts":"1680807434871"} 2023-04-06 18:57:14,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-06 18:57:14,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 in 190 msec 2023-04-06 18:57:14,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-06 18:57:14,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, ASSIGN in 351 msec 2023-04-06 18:57:14,878 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:57:14,878 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807434878"}]},"ts":"1680807434878"} 2023-04-06 18:57:14,879 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:57:14,881 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:57:14,883 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2070 sec 2023-04-06 18:57:15,529 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-06 18:57:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 18:57:15,787 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-06 18:57:15,789 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x12394d27 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,797 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7187fcf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,798 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,800 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38500, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,802 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:57:15,805 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:57:15,807 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x29a548ac to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,812 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@712d78ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,812 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x43184c57 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,823 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@407b9cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,824 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3efec6 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,827 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f456fa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,827 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,831 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,832 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,843 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,843 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x57d25552 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,852 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bae1a64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,852 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x13695730 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,859 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5654f8ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,860 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x0993cc7d to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,865 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@509adaa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,865 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x69b9b3fd to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,876 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e5e7a93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,877 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x1656b283 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:15,884 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3254ddef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:15,890 DEBUG [hconnection-0x543ea65b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,891 DEBUG [hconnection-0x6dd98eb6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,891 DEBUG [hconnection-0x15361d37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:15,891 DEBUG [hconnection-0x24915838-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,892 DEBUG [hconnection-0x495512ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,893 DEBUG [hconnection-0x74b22eab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,894 DEBUG [hconnection-0x3c98ad3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,894 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,894 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,894 DEBUG [hconnection-0x77fc8f6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,895 DEBUG [hconnection-0x3f99f635-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,897 DEBUG [hconnection-0x35a50502-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:15,901 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38604, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38558, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,902 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,908 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:15,908 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:15,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:15,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:15,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:15,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:15,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:15,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:15,915 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:15,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:15,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:15,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:57:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:15,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:15,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:15,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:15,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:15,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:15,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:15,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:15,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:15,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:15,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:15,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:15,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:15,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:15,926 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:15,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:15,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:15,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:15,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e88fa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:15,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:15,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:15,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:15,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:15,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:15,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:15,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:15,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807435947, max timestamp=1680807435971Num uniques 859; 2023-04-06 18:57:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:15,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807435947, max timestamp=1680807435971Num uniques 859; 2023-04-06 18:57:15,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807435947, max timestamp=1680807435971Num uniques 859; 2023-04-06 18:57:15,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:15,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:15,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:15,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807495987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807495987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807495990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:15,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807495991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:15,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:15,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807495992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:16,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:16,035 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:16,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:16,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:16,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:16,035 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:16,036 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:16,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:16,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,039 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:16,039 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807496094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807496093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807496093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807496096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807496096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,139 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:16,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:16,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:16,142 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:16,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:16,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:16,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,144 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,144 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:16,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:16,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:16,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:16,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:16,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:16,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:16,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:16,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:16,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:16,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e007bf4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:16,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:16,148 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:16,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:16,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:16,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:16,151 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:16,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:16,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:16,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:16,152 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:16,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,154 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:16,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:16,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,156 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:16,156 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:16,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,256 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:16,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:16,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:16,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:16,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:16,259 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:16,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:16,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:16,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:16,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:16,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:16,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:16,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:16,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:16,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:16,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:16,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:16,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,263 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:16,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:16,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0b6071[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:16,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:16,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:16,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:16,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:16,267 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:16,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:16,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:16,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:16,267 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:16,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,272 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:16,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,272 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807496296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807496296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807496297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807496299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807496300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/0febdde148ef43c59f4a5efb992f6a83 2023-04-06 18:57:16,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/a458b31a391d4760a9efac18b67c0689 2023-04-06 18:57:16,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/46624752d2f24c54a21c45f52e6456c1 2023-04-06 18:57:16,372 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:16,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:16,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:16,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:16,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:16,375 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:16,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:16,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:16,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/0febdde148ef43c59f4a5efb992f6a83 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83 2023-04-06 18:57:16,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:16,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:16,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:16,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:16,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:16,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:16,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:16,379 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:16,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:16,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:16,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:16,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:16,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:16,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:16,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 18:57:16,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/a458b31a391d4760a9efac18b67c0689 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689 2023-04-06 18:57:16,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:16,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 18:57:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/46624752d2f24c54a21c45f52e6456c1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1 2023-04-06 18:57:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:16,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354a93bf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:16,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:16,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:16,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:16,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:16,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:16,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:16,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 18:57:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for dd972e696d2dd9f5c10736fe54de15bd in 474ms, sequenceid=13, compaction requested=false 2023-04-06 18:57:16,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:16,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:16,396 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:16,396 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:16,396 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:16,396 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:16,396 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:16,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:16,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:16,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:16,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:16,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:16,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:16,401 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:16,402 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:16,502 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:16,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:16,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:16,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:16,504 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:16,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:16,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:16,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:16,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:16,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:16,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:16,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:16,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:16,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:16,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:16,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:16,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:16,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:16,507 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:16,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:16,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:16,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:16,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807496607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807496608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807496608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807496609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807496609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807496711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807496711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807496711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807496711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807496712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,916 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7be97a63ddf04f99bca3c461b9f0c794 2023-04-06 18:57:16,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807496914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807496914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807496914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807496914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:16,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:16,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807496914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807497218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807497218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807497218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807497218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807497218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,326 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/b58aadbc8d54453c9425aec2ece78e3f 2023-04-06 18:57:17,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807497721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807497721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807497723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807497724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:17,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807497724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,736 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/c5bb21aa3d2346438de37e37f42d98f6 2023-04-06 18:57:17,740 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7be97a63ddf04f99bca3c461b9f0c794 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794 2023-04-06 18:57:17,743 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794, entries=150, sequenceid=47, filesize=11.6 K 2023-04-06 18:57:17,743 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/b58aadbc8d54453c9425aec2ece78e3f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f 2023-04-06 18:57:17,747 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f, entries=150, sequenceid=47, filesize=11.6 K 2023-04-06 18:57:17,748 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/c5bb21aa3d2346438de37e37f42d98f6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6 2023-04-06 18:57:17,751 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6, entries=150, sequenceid=47, filesize=11.6 K 2023-04-06 18:57:17,751 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=67.09 KB/68700 for dd972e696d2dd9f5c10736fe54de15bd in 1244ms, sequenceid=47, compaction requested=false 2023-04-06 18:57:17,751 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:17,752 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:17,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:17,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:17,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:17,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416714b4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:17,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:17,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,755 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:17,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:17,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:17,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:17,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:17,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:17,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:17,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:17,758 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:17,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:17,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:17,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:17,759 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:17,759 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,759 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,759 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:17,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:17,760 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,761 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:17,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:17,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:17,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:17,763 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:17,763 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:17,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:17,863 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:17,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:17,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:17,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:17,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:17,866 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:17,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:17,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:17,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:17,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:17,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:17,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:17,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:17,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:17,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:17,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:17,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:17,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:17,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:17,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:17,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:17,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:17,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:17,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:17,869 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:17,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:18,277 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/af75cb7dffdd4ec6ab3ba0c9377b626d 2023-04-06 18:57:18,504 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:57:18,685 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/60112e7b87244946ba5f8db94544cb33 2023-04-06 18:57:18,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:18,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:18,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807498734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807498735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807498736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807498736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807498737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807498838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807498838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807498839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807498839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:18,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:18,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807498839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807499040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807499040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807499042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807499042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807499043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,095 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/106f2c8671b94f90b1fcfd6c0fd6e74f 2023-04-06 18:57:19,100 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/af75cb7dffdd4ec6ab3ba0c9377b626d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d 2023-04-06 18:57:19,104 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d, entries=150, sequenceid=60, filesize=11.6 K 2023-04-06 18:57:19,105 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/60112e7b87244946ba5f8db94544cb33 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33 2023-04-06 18:57:19,109 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33, entries=150, sequenceid=60, filesize=11.6 K 2023-04-06 18:57:19,109 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/106f2c8671b94f90b1fcfd6c0fd6e74f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f 2023-04-06 18:57:19,113 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f, entries=150, sequenceid=60, filesize=11.6 K 2023-04-06 18:57:19,114 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for dd972e696d2dd9f5c10736fe54de15bd in 1245ms, sequenceid=60, compaction requested=true 2023-04-06 18:57:19,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:19,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:19,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:19,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,116 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:19,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:19,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0508b7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:19,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:19,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:19,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:19,119 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:19,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:19,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:19,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:19,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:19,121 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:19,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:19,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:19,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:19,121 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:19,122 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,122 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:19,122 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:19,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:19,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,124 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:19,126 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:19,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:19,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:19,127 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:19,127 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:19,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:19,227 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:19,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:19,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:19,229 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:19,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:19,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:19,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:19,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:19,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:19,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:19,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:19,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:19,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:19,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:19,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:19,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:19,232 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:19,232 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:19,344 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:19,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807499348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807499349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807499349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807499351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807499351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807499452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807499452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807499452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807499454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807499454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,639 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/5c0cd848555040d0b73696a62c223bd2 2023-04-06 18:57:19,648 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/c5427262503e43b0969caa486a7f0005 2023-04-06 18:57:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807499653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807499655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807499656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807499656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807499657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807499958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807499959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807499959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807499961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:19,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:19,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807499961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,064 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/211cf4a7890a4a38a9edad33f5ef8b47 2023-04-06 18:57:20,071 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/5c0cd848555040d0b73696a62c223bd2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2 2023-04-06 18:57:20,074 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2, entries=150, sequenceid=79, filesize=11.6 K 2023-04-06 18:57:20,075 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/c5427262503e43b0969caa486a7f0005 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005 2023-04-06 18:57:20,078 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005, entries=150, sequenceid=79, filesize=11.6 K 2023-04-06 18:57:20,079 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/211cf4a7890a4a38a9edad33f5ef8b47 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47 2023-04-06 18:57:20,082 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47, entries=150, sequenceid=79, filesize=11.6 K 2023-04-06 18:57:20,083 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for dd972e696d2dd9f5c10736fe54de15bd in 851ms, sequenceid=79, compaction requested=true 2023-04-06 18:57:20,083 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:20,083 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:20,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:20,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:20,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:20,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:20,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,084 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:20,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:20,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3ce7ae[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:20,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:20,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:20,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:20,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:20,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:20,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:20,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:20,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:20,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:20,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:20,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:20,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:20,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:20,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:20,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:20,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:20,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:20,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:20,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:20,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:20,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:20,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:20,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,094 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:20,094 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:20,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:20,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:20,194 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:20,197 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:20,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:20,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:20,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:20,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:20,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:20,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:20,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:20,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:20,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:20,199 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:20,199 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:20,199 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:20,199 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:20,200 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:20,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:20,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:20,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807500475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807500475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807500478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807500478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807500478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807500579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807500579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807500582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807500582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807500582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,611 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7ec6b6ed164947a590a48df2accb05de 2023-04-06 18:57:20,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807500782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807500783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807500786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807500787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:20,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807500787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,020 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/2f633896275345ebb282898930451be8 2023-04-06 18:57:21,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807501085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807501086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807501088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807501089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807501091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,431 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/407ec92bbf6441a4ae53ef14c53c1b49 2023-04-06 18:57:21,435 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7ec6b6ed164947a590a48df2accb05de as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de 2023-04-06 18:57:21,438 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de, entries=150, sequenceid=92, filesize=11.6 K 2023-04-06 18:57:21,440 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/2f633896275345ebb282898930451be8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8 2023-04-06 18:57:21,443 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8, entries=150, sequenceid=92, filesize=11.6 K 2023-04-06 18:57:21,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/407ec92bbf6441a4ae53ef14c53c1b49 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49 2023-04-06 18:57:21,447 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49, entries=150, sequenceid=92, filesize=11.6 K 2023-04-06 18:57:21,448 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for dd972e696d2dd9f5c10736fe54de15bd in 1249ms, sequenceid=92, compaction requested=true 2023-04-06 18:57:21,448 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:21,448 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:21,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:21,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8a52dd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:21,450 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:21,450 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:21,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:21,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:21,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:21,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:21,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:21,454 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:21,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:21,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:21,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:21,455 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:21,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:21,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:21,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:21,459 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:21,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:21,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:21,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:21,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,460 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:21,460 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:21,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:21,560 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:21,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:21,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:21,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:21,562 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:21,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:21,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:21,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:21,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:21,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,564 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:21,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:21,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:21,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:21,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:21,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:21,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:21,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:21,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:21,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:21,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:21,566 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:21,566 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:21,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:21,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:21,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807501597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807501597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807501598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807501599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807501602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807501708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807501708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807501708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807501708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807501708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807501912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807501913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807501913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807501914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:21,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807501914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:21,973 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/5f70392413e649f69fbe5acfada98208 2023-04-06 18:57:22,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807502214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807502216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807502217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807502217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807502219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,384 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/d389ddf21c804f4990987aeba0a29d62 2023-04-06 18:57:22,393 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/2653a66b85d847bc828f845471eea1d7 2023-04-06 18:57:22,397 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/5f70392413e649f69fbe5acfada98208 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208 2023-04-06 18:57:22,400 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208, entries=150, sequenceid=112, filesize=11.6 K 2023-04-06 18:57:22,400 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/d389ddf21c804f4990987aeba0a29d62 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62 2023-04-06 18:57:22,403 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62, entries=150, sequenceid=112, filesize=11.6 K 2023-04-06 18:57:22,404 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/2653a66b85d847bc828f845471eea1d7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7 2023-04-06 18:57:22,407 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7, entries=150, sequenceid=112, filesize=11.6 K 2023-04-06 18:57:22,408 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for dd972e696d2dd9f5c10736fe54de15bd in 843ms, sequenceid=112, compaction requested=true 2023-04-06 18:57:22,408 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:22,408 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:22,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:22,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:22,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:22,408 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:22,409 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,410 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,410 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:22,410 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:22,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:22,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:22,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55b1214c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:22,412 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:22,412 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,412 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,412 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:22,412 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:22,412 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:22,413 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,413 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:22,413 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:22,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:22,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:22,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:22,416 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:22,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:22,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:22,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:22,416 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:22,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:22,418 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:22,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:22,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:22,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:22,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:22,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,421 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:22,421 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:22,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:22,521 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:22,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:22,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:22,526 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:22,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:22,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:22,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:22,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:22,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:22,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:22,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:22,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:22,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:22,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:22,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:22,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:22,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:22,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:22,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:22,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:22,529 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:22,529 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:22,530 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:22,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:22,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:22,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:22,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807502735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807502738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807502738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807502740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807502739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807502841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807502843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807502843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807502846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:22,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807502847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:22,944 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/9697c63756f84f5dbe12d157c2390a16 2023-04-06 18:57:22,965 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/d9ed5af582fe47d0aeea14044ea9adc4 2023-04-06 18:57:23,001 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/acad90bfcdf849daac5a6a2fd3702704 2023-04-06 18:57:23,014 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/9697c63756f84f5dbe12d157c2390a16 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16 2023-04-06 18:57:23,018 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16, entries=150, sequenceid=124, filesize=11.6 K 2023-04-06 18:57:23,019 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/d9ed5af582fe47d0aeea14044ea9adc4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4 2023-04-06 18:57:23,025 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4, entries=150, sequenceid=124, filesize=11.6 K 2023-04-06 18:57:23,026 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/acad90bfcdf849daac5a6a2fd3702704 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704 2023-04-06 18:57:23,030 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704, entries=150, sequenceid=124, filesize=11.6 K 2023-04-06 18:57:23,031 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for dd972e696d2dd9f5c10736fe54de15bd in 502ms, sequenceid=124, compaction requested=true 2023-04-06 18:57:23,031 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:23,031 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:23,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:23,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:23,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:23,032 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:23,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a5cbfd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:23,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:23,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:23,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:23,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:23,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:23,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:23,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,038 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,039 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:23,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:23,041 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:23,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:23,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:23,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:23,041 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:23,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,044 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,045 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:23,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:23,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:23,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,049 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:23,049 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,049 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:23,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807442734, max timestamp=1680807443050Num uniques 859; 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807442734, max timestamp=1680807443050Num uniques 859; 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807442734, max timestamp=1680807443050Num uniques 859; 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,053 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-06 18:57:23,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:23,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:23,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:23,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/9bd97e0000494625823c199fde76f88e 2023-04-06 18:57:23,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807443068, max timestamp=1680807443088Num uniques 859; 2023-04-06 18:57:23,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807443068, max timestamp=1680807443090Num uniques 859; 2023-04-06 18:57:23,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807443068, max timestamp=1680807443090Num uniques 859; 2023-04-06 18:57:23,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:23,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/aae71273857e4076b812b70ab31a3c6c 2023-04-06 18:57:23,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807443111, max timestamp=1680807443120Num uniques 1018; 2023-04-06 18:57:23,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807443111, max timestamp=1680807443120Num uniques 1018; 2023-04-06 18:57:23,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:23,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:23,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807443111, max timestamp=1680807443120Num uniques 1018; 2023-04-06 18:57:23,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:23,149 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:23,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:23,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807443144, max timestamp=1680807443151Num uniques 1027; 2023-04-06 18:57:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:57:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807443144, max timestamp=1680807443151Num uniques 1027; 2023-04-06 18:57:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:57:23,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807443144, max timestamp=1680807443151Num uniques 1027; 2023-04-06 18:57:23,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:57:23,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:23,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:23,159 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:23,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:23,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:23,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:23,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:23,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:23,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:23,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:23,163 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,163 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:23,163 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:23,163 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:23,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:23,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,164 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:23,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0e4df0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:23,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:23,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:23,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:23,167 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,167 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:23,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:23,168 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,168 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:23,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:23,170 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:23,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:23,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:23,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:23,170 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:23,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:23,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:23,176 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:23,176 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:23,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807443179, max timestamp=1680807443187Num uniques 1036; 2023-04-06 18:57:23,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 18:57:23,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807443179, max timestamp=1680807443187Num uniques 1086; 2023-04-06 18:57:23,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807443179, max timestamp=1680807443187Num uniques 1086; 2023-04-06 18:57:23,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 18:57:23,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 18:57:23,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807503217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807443207, max timestamp=1680807443226Num uniques 1095; 2023-04-06 18:57:23,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-06 18:57:23,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807443207, max timestamp=1680807443226Num uniques 1095; 2023-04-06 18:57:23,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-06 18:57:23,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807443207, max timestamp=1680807443226Num uniques 1095; 2023-04-06 18:57:23,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-06 18:57:23,276 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:23,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:23,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:23,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:23,288 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:23,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:23,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:23,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,289 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,289 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:23,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:23,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:23,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:23,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:23,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:23,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807503287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807503287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807503289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:23,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,296 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:23,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:23,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:23,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:23,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:23,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:23,308 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,309 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:23,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:23,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556ceef6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:23,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:23,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:23,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:23,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:23,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,316 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:23,316 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,316 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:23,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:23,318 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:23,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:23,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:23,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:23,318 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:23,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,319 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,321 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,326 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:23,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:23,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,326 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:23,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807503331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807503397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807503397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807503397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,426 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:23,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:23,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:23,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:23,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:23,429 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:23,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:23,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:23,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,431 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:23,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:23,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:23,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:23,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:23,433 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:23,433 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,434 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98b1e56[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:23,435 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:23,435 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,435 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:23,436 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,436 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:23,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,437 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,437 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:23,438 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:23,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:23,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:23,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:23,438 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:23,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:23,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:23,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:23,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:23,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:23,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,443 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:23,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:23,443 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:23,443 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:23,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:23,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/b1d7dd1660c6446ea28752941b081645 2023-04-06 18:57:23,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/9bd97e0000494625823c199fde76f88e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e 2023-04-06 18:57:23,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e, entries=300, sequenceid=147, filesize=18.8 K 2023-04-06 18:57:23,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/aae71273857e4076b812b70ab31a3c6c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c 2023-04-06 18:57:23,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807503536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 18:57:23,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/b1d7dd1660c6446ea28752941b081645 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645 2023-04-06 18:57:23,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 18:57:23,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=154.31 KB/158010 for dd972e696d2dd9f5c10736fe54de15bd in 489ms, sequenceid=147, compaction requested=true 2023-04-06 18:57:23,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:23,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:23,542 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 18:57:23,542 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 18:57:23,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:23,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:23,543 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:23,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:23,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:23,545 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 18:57:23,545 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/B is initiating minor compaction (all files) 2023-04-06 18:57:23,545 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/B in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,545 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 102408 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 18:57:23,545 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=93.0 K 2023-04-06 18:57:23,545 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/A is initiating minor compaction (all files) 2023-04-06 18:57:23,545 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/A in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,545 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=100.0 K 2023-04-06 18:57:23,546 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0febdde148ef43c59f4a5efb992f6a83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807435916 2023-04-06 18:57:23,546 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a458b31a391d4760a9efac18b67c0689, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807435916 2023-04-06 18:57:23,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:23,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:23,546 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:23,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,546 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7be97a63ddf04f99bca3c461b9f0c794, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807435981 2023-04-06 18:57:23,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:23,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:23,546 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b58aadbc8d54453c9425aec2ece78e3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807435981 2023-04-06 18:57:23,547 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting af75cb7dffdd4ec6ab3ba0c9377b626d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680807436607 2023-04-06 18:57:23,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,547 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 60112e7b87244946ba5f8db94544cb33, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680807436607 2023-04-06 18:57:23,547 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 5c0cd848555040d0b73696a62c223bd2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680807438731 2023-04-06 18:57:23,547 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c5427262503e43b0969caa486a7f0005, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680807438731 2023-04-06 18:57:23,547 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7ec6b6ed164947a590a48df2accb05de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680807439344 2023-04-06 18:57:23,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,547 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:23,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:23,548 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2f633896275345ebb282898930451be8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680807439344 2023-04-06 18:57:23,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,548 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 5f70392413e649f69fbe5acfada98208, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680807440469 2023-04-06 18:57:23,548 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d389ddf21c804f4990987aeba0a29d62, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680807440469 2023-04-06 18:57:23,548 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9697c63756f84f5dbe12d157c2390a16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680807441597 2023-04-06 18:57:23,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:23,549 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d9ed5af582fe47d0aeea14044ea9adc4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680807441597 2023-04-06 18:57:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:23,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:23,553 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting aae71273857e4076b812b70ab31a3c6c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:23,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:23,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:23,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:23,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:23,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:23,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:23,554 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:23,554 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,554 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:23,554 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-06 18:57:23,555 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:23,557 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9bd97e0000494625823c199fde76f88e, keycount=300, bloomtype=ROW, size=18.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:23,557 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:23,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:23,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:23,580 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#B#compaction#231 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:23,584 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#A#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:23,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:23,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:23,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807503606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807503608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,616 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/cbac54ddd4a949d896c37d250e779b63 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/cbac54ddd4a949d896c37d250e779b63 2023-04-06 18:57:23,617 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/20662b67d8274c7084705d4e13b3c643 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/20662b67d8274c7084705d4e13b3c643 2023-04-06 18:57:23,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807503611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,626 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/B of dd972e696d2dd9f5c10736fe54de15bd into 20662b67d8274c7084705d4e13b3c643(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:23,626 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:23,626 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/B, priority=8, startTime=1680807443542; duration=0sec 2023-04-06 18:57:23,626 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:23,626 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 18:57:23,627 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/A of dd972e696d2dd9f5c10736fe54de15bd into cbac54ddd4a949d896c37d250e779b63(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:23,627 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:23,627 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/A, priority=8, startTime=1680807443542; duration=0sec 2023-04-06 18:57:23,627 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:23,628 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95238 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 18:57:23,628 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/C is initiating minor compaction (all files) 2023-04-06 18:57:23,628 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/C in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:23,628 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=93.0 K 2023-04-06 18:57:23,629 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 46624752d2f24c54a21c45f52e6456c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680807435916 2023-04-06 18:57:23,629 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c5bb21aa3d2346438de37e37f42d98f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680807435981 2023-04-06 18:57:23,629 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 106f2c8671b94f90b1fcfd6c0fd6e74f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680807436607 2023-04-06 18:57:23,629 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 211cf4a7890a4a38a9edad33f5ef8b47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680807438731 2023-04-06 18:57:23,630 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 407ec92bbf6441a4ae53ef14c53c1b49, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1680807439344 2023-04-06 18:57:23,630 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2653a66b85d847bc828f845471eea1d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1680807440469 2023-04-06 18:57:23,630 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting acad90bfcdf849daac5a6a2fd3702704, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680807441597 2023-04-06 18:57:23,630 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b1d7dd1660c6446ea28752941b081645, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:23,647 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#C#compaction#233 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:23,676 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/750fb7eef440448b9443bbe3f2ce2241 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/750fb7eef440448b9443bbe3f2ce2241 2023-04-06 18:57:23,682 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/C of dd972e696d2dd9f5c10736fe54de15bd into 750fb7eef440448b9443bbe3f2ce2241(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:23,682 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:23,682 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/C, priority=8, startTime=1680807443542; duration=0sec 2023-04-06 18:57:23,683 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:23,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807503716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807503716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807503719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807503838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807503919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807503919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807503919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807503921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:23,995 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/b79463049e584a76a50310ccbb2e37e2 2023-04-06 18:57:24,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807504222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807504223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807504223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807504224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38620 deadline: 1680807504340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,405 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/178977bb96844dc1b1e1c1f8774549f8 2023-04-06 18:57:24,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38508 deadline: 1680807504725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807504727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38568 deadline: 1680807504727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:24,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38524 deadline: 1680807504729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,814 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/8e0e12d40c214a2ebbe760192e02844f 2023-04-06 18:57:24,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/b79463049e584a76a50310ccbb2e37e2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2 2023-04-06 18:57:24,821 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 18:57:24,822 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/178977bb96844dc1b1e1c1f8774549f8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8 2023-04-06 18:57:24,825 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 18:57:24,826 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/8e0e12d40c214a2ebbe760192e02844f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f 2023-04-06 18:57:24,830 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 18:57:24,830 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for dd972e696d2dd9f5c10736fe54de15bd in 1276ms, sequenceid=253, compaction requested=false 2023-04-06 18:57:24,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:24,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:24,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:24,832 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,832 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:24,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:24,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dec6b9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:24,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:24,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:24,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:24,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:24,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:24,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:24,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:24,838 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:24,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:24,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:24,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:24,838 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:24,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:24,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:24,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:24,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:24,844 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:24,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:24,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:24,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:24,844 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:24,844 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:24,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:24,944 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:24,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:24,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:24,947 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:24,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:24,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:24,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:24,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:24,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:24,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:24,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:24,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:24,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:24,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:24,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:24,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:24,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:24,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:24,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:24,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:24,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:24,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:24,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:24,950 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:24,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:25,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:25,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:25,372 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/aa757276b31c42b79329d74304fc3509 2023-04-06 18:57:25,380 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/84d893ad800640f98e3f3fd191dce0fe 2023-04-06 18:57:25,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807445383, max timestamp=1680807445393Num uniques 859; 2023-04-06 18:57:25,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:25,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807445383, max timestamp=1680807445393Num uniques 859; 2023-04-06 18:57:25,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:25,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807445383, max timestamp=1680807445393Num uniques 859; 2023-04-06 18:57:25,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:25,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807445435, max timestamp=1680807445445Num uniques 1018; 2023-04-06 18:57:25,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807445435, max timestamp=1680807445445Num uniques 1018; 2023-04-06 18:57:25,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:25,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:25,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807445435, max timestamp=1680807445445Num uniques 1018; 2023-04-06 18:57:25,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:25,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807445480, max timestamp=1680807445496Num uniques 1027; 2023-04-06 18:57:25,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:57:25,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807445480, max timestamp=1680807445496Num uniques 1027; 2023-04-06 18:57:25,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:57:25,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807445480, max timestamp=1680807445496Num uniques 1027; 2023-04-06 18:57:25,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807445543, max timestamp=1680807445553Num uniques 1036; 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807445543, max timestamp=1680807445553Num uniques 1036; 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807445543, max timestamp=1680807445553Num uniques 1036; 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:57:25,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:57:25,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807445580, max timestamp=1680807445606Num uniques 1045; 2023-04-06 18:57:25,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:25,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807445580, max timestamp=1680807445606Num uniques 1045; 2023-04-06 18:57:25,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:25,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807445580, max timestamp=1680807445606Num uniques 1045; 2023-04-06 18:57:25,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:25,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807445644, max timestamp=1680807445653Num uniques 1054; 2023-04-06 18:57:25,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807445644, max timestamp=1680807445653Num uniques 1054; 2023-04-06 18:57:25,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807445644, max timestamp=1680807445653Num uniques 1054; 2023-04-06 18:57:25,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 18:57:25,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 18:57:25,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807445688, max timestamp=1680807445700Num uniques 1013; 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807445688, max timestamp=1680807445700Num uniques 1013; 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807445688, max timestamp=1680807445700Num uniques 1013; 2023-04-06 18:57:25,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807445732, max timestamp=1680807445737Num uniques 1022; 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807445732, max timestamp=1680807445737Num uniques 1022; 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807445732, max timestamp=1680807445737Num uniques 1022; 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-06 18:57:25,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-06 18:57:25,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807445750, max timestamp=1680807445756Num uniques 1081; 2023-04-06 18:57:25,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:57:25,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807445750, max timestamp=1680807445756Num uniques 1081; 2023-04-06 18:57:25,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807445750, max timestamp=1680807445756Num uniques 1081; 2023-04-06 18:57:25,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:57:25,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:57:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807505770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807445769, max timestamp=1680807445772Num uniques 1090; 2023-04-06 18:57:25,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:57:25,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807445769, max timestamp=1680807445772Num uniques 1090; 2023-04-06 18:57:25,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:57:25,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807445769, max timestamp=1680807445772Num uniques 1090; 2023-04-06 18:57:25,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:57:25,789 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/d8ff2625cc8149f7ac640dc949c40c9c 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680807445785, max timestamp=1680807445790Num uniques 1049; 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680807445785, max timestamp=1680807445790Num uniques 1049; 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680807445785, max timestamp=1680807445790Num uniques 1049; 2023-04-06 18:57:25,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-06 18:57:25,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/aa757276b31c42b79329d74304fc3509 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509 2023-04-06 18:57:25,800 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 18:57:25,801 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/84d893ad800640f98e3f3fd191dce0fe as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe 2023-04-06 18:57:25,805 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 18:57:25,805 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/d8ff2625cc8149f7ac640dc949c40c9c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c 2023-04-06 18:57:25,810 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 18:57:25,810 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=114.05 KB/116790 for dd972e696d2dd9f5c10736fe54de15bd in 860ms, sequenceid=263, compaction requested=true 2023-04-06 18:57:25,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:25,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:25,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:25,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:25,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:25,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:25,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,811 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe67e5f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:25,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:25,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:25,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=120.76 KB heapSize=336.70 KB 2023-04-06 18:57:25,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:25,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:25,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:25,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:25,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:25,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:25,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:25,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:25,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:25,815 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:25,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:25,818 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:25,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:25,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:25,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:25,818 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:25,819 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,819 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:25,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:25,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:25,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:25,824 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:25,824 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:25,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807505827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807505828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807505828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807505829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807505875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,924 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:25,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:25,927 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:25,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:25,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:25,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:25,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,928 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:25,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:25,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:25,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:25,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:25,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:25,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:25,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:25,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:25,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:25,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcf240b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:25,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:25,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,934 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807505932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:25,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:25,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:25,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:25,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807505934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807505935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:25,937 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:25,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807505935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:25,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:25,939 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:25,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:25,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:25,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:25,939 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:25,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:25,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:25,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:25,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:25,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:25,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:25,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:25,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:25,944 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:25,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:25,944 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:25,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:25,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,044 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,047 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,049 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,049 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,049 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,049 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,050 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,051 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce7ea4a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,052 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,053 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,055 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,055 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,055 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,056 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,056 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,056 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:26,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,060 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,060 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807506079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807506137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807506137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807506137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807506138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,160 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,162 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,166 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,166 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,166 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,166 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5c1c61[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,169 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,169 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,170 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,172 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,172 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,175 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,176 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,177 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,177 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,177 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/39f8e41858644f9fa6e6258808028575 2023-04-06 18:57:26,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/62e6539f3e4e46bb8674b86f9ebfb771 2023-04-06 18:57:26,277 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,279 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,280 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,281 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,282 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,282 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,282 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,282 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,282 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@849733[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,289 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,291 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,291 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,293 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,296 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,296 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,296 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,296 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38526 deadline: 1680807506384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,396 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,399 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,401 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,401 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,401 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,401 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,401 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,402 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,402 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34df0454[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,404 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,405 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,405 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,405 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,407 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,408 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,408 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,408 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,408 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,408 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,409 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,410 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,412 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,412 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,412 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:26,412 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,412 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807506439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807506441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807506442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807506442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,512 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,515 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,518 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,518 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,518 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,518 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,519 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,519 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,519 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,519 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,519 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,520 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,521 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c91d3f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,522 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,523 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,524 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,524 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,524 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,524 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,527 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,527 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,528 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,528 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,528 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,529 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,534 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,534 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,534 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,635 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,637 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:26,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:26,641 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,642 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,642 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:26,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5ab4b3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:26,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:26,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,644 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:26,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:26,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:26,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:26,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:26,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:26,651 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/b3921224d5f44e0bbc68fdbbbb180197 2023-04-06 18:57:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:26,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:26,655 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:26,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:26,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:26,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:26,655 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:26,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/39f8e41858644f9fa6e6258808028575 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575 2023-04-06 18:57:26,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575, entries=250, sequenceid=470, filesize=16.7 K 2023-04-06 18:57:26,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/62e6539f3e4e46bb8674b86f9ebfb771 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771 2023-04-06 18:57:26,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:26,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:26,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,667 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771, entries=150, sequenceid=470, filesize=11.9 K 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:26,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:26,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:26,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,673 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:26,674 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:26,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:26,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:26,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/b3921224d5f44e0bbc68fdbbbb180197 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197 2023-04-06 18:57:26,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197, entries=150, sequenceid=470, filesize=11.9 K 2023-04-06 18:57:26,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~376.50 KB/385536, currentSize=53.67 KB/54960 for dd972e696d2dd9f5c10736fe54de15bd in 866ms, sequenceid=470, compaction requested=true 2023-04-06 18:57:26,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:26,681 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:26,681 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:26,682 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53548 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:26,682 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/A is initiating minor compaction (all files) 2023-04-06 18:57:26,682 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/A in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,683 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/cbac54ddd4a949d896c37d250e779b63, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=52.3 K 2023-04-06 18:57:26,683 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48668 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:26,683 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/B is initiating minor compaction (all files) 2023-04-06 18:57:26,683 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/B in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,683 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/20662b67d8274c7084705d4e13b3c643, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=47.5 K 2023-04-06 18:57:26,683 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting cbac54ddd4a949d896c37d250e779b63, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:26,683 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 20662b67d8274c7084705d4e13b3c643, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:26,684 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b79463049e584a76a50310ccbb2e37e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807443235 2023-04-06 18:57:26,684 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 178977bb96844dc1b1e1c1f8774549f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807443235 2023-04-06 18:57:26,684 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting aa757276b31c42b79329d74304fc3509, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680807443605 2023-04-06 18:57:26,684 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 84d893ad800640f98e3f3fd191dce0fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680807443605 2023-04-06 18:57:26,685 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 39f8e41858644f9fa6e6258808028575, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445803 2023-04-06 18:57:26,685 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 62e6539f3e4e46bb8674b86f9ebfb771, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445807 2023-04-06 18:57:26,704 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#A#compaction#242 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:26,712 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#B#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:26,751 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/540b7f1ca96f47b686d1f146a52971d0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/540b7f1ca96f47b686d1f146a52971d0 2023-04-06 18:57:26,758 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/A of dd972e696d2dd9f5c10736fe54de15bd into 540b7f1ca96f47b686d1f146a52971d0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:26,758 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:26,758 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/A, priority=12, startTime=1680807446681; duration=0sec 2023-04-06 18:57:26,758 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:26,758 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:26,759 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48668 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:26,760 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/C is initiating minor compaction (all files) 2023-04-06 18:57:26,760 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/C in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,760 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/750fb7eef440448b9443bbe3f2ce2241, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=47.5 K 2023-04-06 18:57:26,760 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 750fb7eef440448b9443bbe3f2ce2241, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680807442734 2023-04-06 18:57:26,760 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 8e0e12d40c214a2ebbe760192e02844f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807443235 2023-04-06 18:57:26,761 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d8ff2625cc8149f7ac640dc949c40c9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680807443605 2023-04-06 18:57:26,761 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b3921224d5f44e0bbc68fdbbbb180197, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445807 2023-04-06 18:57:26,767 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:26,774 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:26,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:26,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:26,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:26,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:26,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:26,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,778 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:26,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:26,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:26,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:26,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:26,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:26,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:26,779 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:26,779 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:26,779 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:57:26,779 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:26,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:26,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:26,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:26,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:26,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:26,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:26,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807446933, max timestamp=1680807446943Num uniques 859; 2023-04-06 18:57:26,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807446933, max timestamp=1680807446943Num uniques 859; 2023-04-06 18:57:26,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:26,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807446933, max timestamp=1680807446943Num uniques 859; 2023-04-06 18:57:26,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:26,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:26,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807506955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807506958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807506959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807506960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:26,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807506959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807507061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807507064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807507064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807507065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807507065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,153 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/9792fea5d6054eb0a1c457b26edfbde4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9792fea5d6054eb0a1c457b26edfbde4 2023-04-06 18:57:27,162 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/B of dd972e696d2dd9f5c10736fe54de15bd into 9792fea5d6054eb0a1c457b26edfbde4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:27,162 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:27,162 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/B, priority=12, startTime=1680807446681; duration=0sec 2023-04-06 18:57:27,163 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:27,175 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/f05ce3bd1d4a4df598c5350fc61e6f99 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/f05ce3bd1d4a4df598c5350fc61e6f99 2023-04-06 18:57:27,179 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/C of dd972e696d2dd9f5c10736fe54de15bd into f05ce3bd1d4a4df598c5350fc61e6f99(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:27,179 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:27,179 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/C, priority=12, startTime=1680807446681; duration=0sec 2023-04-06 18:57:27,179 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:27,188 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7a0409c647a2410f88b4850e7f874cf4 2023-04-06 18:57:27,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807507263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807507267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807507267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807507268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807507268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807507566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807507570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807507571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807507572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:27,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807507573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:27,606 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/544f567b5dd747229d6dce7e988bdf1f 2023-04-06 18:57:28,015 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/e14589b68db54a52af31633156be99f9 2023-04-06 18:57:28,020 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/7a0409c647a2410f88b4850e7f874cf4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4 2023-04-06 18:57:28,023 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4, entries=150, sequenceid=482, filesize=11.9 K 2023-04-06 18:57:28,024 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/544f567b5dd747229d6dce7e988bdf1f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f 2023-04-06 18:57:28,027 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f, entries=150, sequenceid=482, filesize=11.9 K 2023-04-06 18:57:28,028 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/e14589b68db54a52af31633156be99f9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9 2023-04-06 18:57:28,031 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9, entries=150, sequenceid=482, filesize=11.9 K 2023-04-06 18:57:28,032 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for dd972e696d2dd9f5c10736fe54de15bd in 1253ms, sequenceid=482, compaction requested=false 2023-04-06 18:57:28,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:28,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,033 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,034 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4381e4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,036 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,039 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,039 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,042 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,042 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,043 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,043 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,043 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,043 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:28,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:57:28,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:28,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:28,072 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:28,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:28,072 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:28,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:28,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/77895d0ddf364fd0bef1065a0871281c 2023-04-06 18:57:28,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807508078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807508079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807508080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807508080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807508081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,143 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,146 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,148 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,148 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,148 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,149 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f2dbfc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,159 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,160 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807508183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807508183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807508184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807508184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807508184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,260 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,262 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,264 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,264 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,266 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8d1458[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,270 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,271 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,275 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,275 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,375 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,378 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,379 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,379 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,379 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,379 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,381 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,381 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,381 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,381 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,382 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629cc656[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,388 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807508386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,388 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807508387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807508387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807508388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807508388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,394 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,394 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/00ed24f1993b406e8f4ded06a0bdba07 2023-04-06 18:57:28,494 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,502 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,502 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,505 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d33740[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,518 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,518 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,618 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,621 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,622 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,623 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,624 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,624 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,625 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243b3dbd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,630 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,630 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,630 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,631 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,631 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,631 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,633 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,633 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,634 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,639 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,639 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,639 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807508690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807508690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807508691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807508692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807508694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,740 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,766 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,766 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,767 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,767 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,767 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,818 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4070c2b6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,823 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,823 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,823 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,824 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,824 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,830 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,834 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,835 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,835 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/4e1e6ee05f0549ef86dd2e70d5658369 2023-04-06 18:57:28,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/77895d0ddf364fd0bef1065a0871281c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c 2023-04-06 18:57:28,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c, entries=200, sequenceid=521, filesize=14.3 K 2023-04-06 18:57:28,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/00ed24f1993b406e8f4ded06a0bdba07 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07 2023-04-06 18:57:28,935 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:28,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:28,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07, entries=150, sequenceid=521, filesize=11.9 K 2023-04-06 18:57:28,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/4e1e6ee05f0549ef86dd2e70d5658369 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369 2023-04-06 18:57:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:28,954 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:28,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:28,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:28,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:28,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:28,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:28,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:28,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:28,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:28,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:28,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:28,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:28,960 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,960 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:28,960 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:28,960 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:28,960 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:28,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:28,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da1298[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:28,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:28,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:28,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:28,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369, entries=150, sequenceid=521, filesize=11.9 K 2023-04-06 18:57:28,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for dd972e696d2dd9f5c10736fe54de15bd in 897ms, sequenceid=521, compaction requested=true 2023-04-06 18:57:28,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:28,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:28,968 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:28,969 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:28,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:28,969 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:28,970 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39408 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:28,970 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/A is initiating minor compaction (all files) 2023-04-06 18:57:28,970 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/A in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,971 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/540b7f1ca96f47b686d1f146a52971d0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=38.5 K 2023-04-06 18:57:28,971 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 540b7f1ca96f47b686d1f146a52971d0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445807 2023-04-06 18:57:28,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,974 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:28,975 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/B is initiating minor compaction (all files) 2023-04-06 18:57:28,975 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/B in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:28,975 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9792fea5d6054eb0a1c457b26edfbde4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=36.1 K 2023-04-06 18:57:28,976 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9792fea5d6054eb0a1c457b26edfbde4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445807 2023-04-06 18:57:28,974 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7a0409c647a2410f88b4850e7f874cf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680807445820 2023-04-06 18:57:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:28,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:28,979 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:28,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:28,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:28,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:28,979 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:28,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:28,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,982 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 544f567b5dd747229d6dce7e988bdf1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680807445820 2023-04-06 18:57:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:28,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,984 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:28,990 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 77895d0ddf364fd0bef1065a0871281c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446953 2023-04-06 18:57:28,990 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 00ed24f1993b406e8f4ded06a0bdba07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446956 2023-04-06 18:57:28,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:28,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:28,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:28,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:28,993 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:28,994 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:28,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:29,023 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:29,071 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:29,094 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:29,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:29,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:29,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:29,103 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:29,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:29,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:29,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:29,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:29,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:29,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:29,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:29,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:29,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:29,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:29,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:29,158 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:57:29,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:29,158 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:29,159 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:29,159 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:29,159 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:29,172 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/48518a6b3d8049a0b3e8dadd34d6439e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/48518a6b3d8049a0b3e8dadd34d6439e 2023-04-06 18:57:29,177 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/B of dd972e696d2dd9f5c10736fe54de15bd into 48518a6b3d8049a0b3e8dadd34d6439e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:29,177 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:29,178 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/B, priority=13, startTime=1680807448968; duration=0sec 2023-04-06 18:57:29,178 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:29,178 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:29,187 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:29,187 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/C is initiating minor compaction (all files) 2023-04-06 18:57:29,188 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/C in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:29,188 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/f05ce3bd1d4a4df598c5350fc61e6f99, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=36.1 K 2023-04-06 18:57:29,188 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f05ce3bd1d4a4df598c5350fc61e6f99, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680807445807 2023-04-06 18:57:29,188 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e14589b68db54a52af31633156be99f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1680807445820 2023-04-06 18:57:29,189 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4e1e6ee05f0549ef86dd2e70d5658369, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446956 2023-04-06 18:57:29,202 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/36e02eb297f2457189da148235089f52 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/36e02eb297f2457189da148235089f52 2023-04-06 18:57:29,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:29,208 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:29,225 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/A of dd972e696d2dd9f5c10736fe54de15bd into 36e02eb297f2457189da148235089f52(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:29,225 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:29,225 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/A, priority=13, startTime=1680807448968; duration=0sec 2023-04-06 18:57:29,226 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:29,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680807449226, max timestamp=1680807449233Num uniques 859; 2023-04-06 18:57:29,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=10346, suffix heap size=117064,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=226 2023-04-06 18:57:29,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680807449226, max timestamp=1680807449233Num uniques 859; 2023-04-06 18:57:29,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 18:57:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680807449226, max timestamp=1680807449233Num uniques 859; 2023-04-06 18:57:29,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 18:57:29,243 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:29,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807449251, max timestamp=1680807449275Num uniques 1085; 2023-04-06 18:57:29,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=12268, suffix heap size=140808,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=268 2023-04-06 18:57:29,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807449251, max timestamp=1680807449275Num uniques 1009; 2023-04-06 18:57:29,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-06 18:57:29,278 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/c36b0b957bb1412fb02f7074d28e4f96 2023-04-06 18:57:29,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807449251, max timestamp=1680807449275Num uniques 1009; 2023-04-06 18:57:29,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-06 18:57:29,297 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/b903c60aa3664b54a5b6e864127380d7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b903c60aa3664b54a5b6e864127380d7 2023-04-06 18:57:29,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807509312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807509315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807509311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807509317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807509321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,343 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/C of dd972e696d2dd9f5c10736fe54de15bd into b903c60aa3664b54a5b6e864127380d7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:29,343 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:29,343 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/C, priority=13, startTime=1680807448969; duration=0sec 2023-04-06 18:57:29,343 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:29,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807509422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807509422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807509423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807509427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807509428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807509627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807509627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807509630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807509630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807509638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,746 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/e36d7a127fea4405b4c1f6d13a72304e 2023-04-06 18:57:29,771 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/9f5eae5df207409b84dd3400b6152433 2023-04-06 18:57:29,775 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/c36b0b957bb1412fb02f7074d28e4f96 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96 2023-04-06 18:57:29,778 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96, entries=150, sequenceid=531, filesize=11.9 K 2023-04-06 18:57:29,779 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/e36d7a127fea4405b4c1f6d13a72304e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e 2023-04-06 18:57:29,783 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e, entries=150, sequenceid=531, filesize=11.9 K 2023-04-06 18:57:29,784 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/9f5eae5df207409b84dd3400b6152433 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433 2023-04-06 18:57:29,805 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433, entries=150, sequenceid=531, filesize=11.9 K 2023-04-06 18:57:29,806 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for dd972e696d2dd9f5c10736fe54de15bd in 648ms, sequenceid=531, compaction requested=false 2023-04-06 18:57:29,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:29,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:29,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:29,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:29,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:29,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:29,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,807 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccb4474[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:29,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:29,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:29,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:29,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:29,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:29,810 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:29,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:29,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:29,813 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:29,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:29,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:29,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:29,813 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:29,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:29,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:29,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:29,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:29,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:29,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,817 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:29,817 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:29,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:29,917 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:29,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:29,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:29,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:29,926 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:29,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:29,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:29,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:29,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:29,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:29,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:29,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=149.83 KB heapSize=425.08 KB 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:29,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807509938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807509939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:29,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807509941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807509942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807509943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:29,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:29,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:29,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:29,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:29,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:29,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:29,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/6b9e6b06bc324d188928ab672fb505d4 2023-04-06 18:57:29,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:29,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:29,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:29,965 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:29,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:29,966 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:29,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e04085[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:29,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:29,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:29,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:29,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:29,971 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:29,971 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:29,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:29,973 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:29,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:29,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:29,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:29,973 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:29,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:29,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:29,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:29,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:29,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:29,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:29,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:29,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:29,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:29,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:29,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:29,995 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:29,995 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:29,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:29,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:29,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:29,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807510049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807510051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,098 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:30,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:30,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:30,106 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:30,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:30,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:30,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:30,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:30,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:30,112 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:30,112 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:30,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:30,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:30,114 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:30,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:30,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:30,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:30,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:30,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b3bae8d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:30,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:30,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:30,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:30,125 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:30,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:30,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:30,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:30,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:30,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:30,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:30,130 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:30,130 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:30,130 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:30,130 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:30,130 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:30,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:30,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,140 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:30,140 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:30,240 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:30,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:30,245 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:30,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:30,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:30,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:30,248 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,248 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:30,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:30,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:30,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:30,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:30,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:30,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:30,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,250 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:30,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:30,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7434f7d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:30,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,252 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:30,252 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:30,252 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:30,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807510250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:30,254 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:30,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807510255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:30,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:30,263 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:30,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:30,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:30,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:30,263 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:30,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,268 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:30,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:30,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:30,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,270 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:30,270 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,270 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,370 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:30,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:30,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:30,374 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:30,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:30,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:30,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,375 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:30,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:30,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:30,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:30,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:30,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:30,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:30,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:30,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:30,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:30,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:30,380 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:30,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:30,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:30,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:30,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:30,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1399d3df[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:30,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:30,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/6526c9fb1f734f229d1460e738da3e45 2023-04-06 18:57:30,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:30,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,398 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:30,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:30,400 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:30,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:30,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:30,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:30,400 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:30,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,408 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,415 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,415 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:30,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:30,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:30,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:30,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:30,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,416 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:30,416 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:30,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/163a67d622a04a7ca414e9731e80545b 2023-04-06 18:57:30,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/6b9e6b06bc324d188928ab672fb505d4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4 2023-04-06 18:57:30,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4, entries=150, sequenceid=589, filesize=11.9 K 2023-04-06 18:57:30,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807510445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/6526c9fb1f734f229d1460e738da3e45 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45 2023-04-06 18:57:30,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807510447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807510449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45, entries=150, sequenceid=589, filesize=11.9 K 2023-04-06 18:57:30,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/163a67d622a04a7ca414e9731e80545b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b 2023-04-06 18:57:30,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b, entries=150, sequenceid=589, filesize=11.9 K 2023-04-06 18:57:30,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~425.03 KB/435232, currentSize=33.54 KB/34350 for dd972e696d2dd9f5c10736fe54de15bd in 533ms, sequenceid=589, compaction requested=true 2023-04-06 18:57:30,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:30,465 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:30,465 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:30,465 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:30,465 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:30,465 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:30,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:30,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/A is initiating minor compaction (all files) 2023-04-06 18:57:30,466 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/A in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,466 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/36e02eb297f2457189da148235089f52, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=36.2 K 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 36e02eb297f2457189da148235089f52, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446956 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/B is initiating minor compaction (all files) 2023-04-06 18:57:30,467 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/B in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,467 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/48518a6b3d8049a0b3e8dadd34d6439e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=36.2 K 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c36b0b957bb1412fb02f7074d28e4f96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680807448077 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 48518a6b3d8049a0b3e8dadd34d6439e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446956 2023-04-06 18:57:30,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6b9e6b06bc324d188928ab672fb505d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:30,468 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e36d7a127fea4405b4c1f6d13a72304e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680807448077 2023-04-06 18:57:30,468 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6526c9fb1f734f229d1460e738da3e45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:30,482 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#A#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:30,483 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#B#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:30,506 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/898947d1e66c4ee5bf92d22a8c606c7a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/898947d1e66c4ee5bf92d22a8c606c7a 2023-04-06 18:57:30,507 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/60b48b6458024fd28cb7e74e05cfaf4e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/60b48b6458024fd28cb7e74e05cfaf4e 2023-04-06 18:57:30,511 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/B of dd972e696d2dd9f5c10736fe54de15bd into 898947d1e66c4ee5bf92d22a8c606c7a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:30,512 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:30,512 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/A of dd972e696d2dd9f5c10736fe54de15bd into 60b48b6458024fd28cb7e74e05cfaf4e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:30,512 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/B, priority=13, startTime=1680807450465; duration=0sec 2023-04-06 18:57:30,512 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:30,512 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/A, priority=13, startTime=1680807450465; duration=0sec 2023-04-06 18:57:30,512 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:30,512 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:30,512 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:30,513 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:30,513 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/C is initiating minor compaction (all files) 2023-04-06 18:57:30,513 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/C in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,513 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b903c60aa3664b54a5b6e864127380d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=36.2 K 2023-04-06 18:57:30,513 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b903c60aa3664b54a5b6e864127380d7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680807446956 2023-04-06 18:57:30,514 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9f5eae5df207409b84dd3400b6152433, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680807448077 2023-04-06 18:57:30,517 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:30,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:30,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:30,530 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 163a67d622a04a7ca414e9731e80545b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:30,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:30,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:30,536 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:30,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:30,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:30,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,543 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:30,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:30,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:30,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:30,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:30,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:30,545 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:30,545 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:30,545 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:30,547 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 18:57:30,547 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:30,547 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:30,547 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:30,547 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:30,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:30,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:30,556 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#C#compaction#263 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:30,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:30,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:30,575 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/3742543e90914fbd84baa236e4aa6bba 2023-04-06 18:57:30,588 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/28804447ed9e4329909b7dba8045a9cf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/28804447ed9e4329909b7dba8045a9cf 2023-04-06 18:57:30,592 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/C of dd972e696d2dd9f5c10736fe54de15bd into 28804447ed9e4329909b7dba8045a9cf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:30,592 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:30,592 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/C, priority=13, startTime=1680807450465; duration=0sec 2023-04-06 18:57:30,593 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:30,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807510605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807510605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807450573, max timestamp=1680807450599Num uniques 859; 2023-04-06 18:57:30,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:30,618 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/0ec7ab78f8194906ad51d459fe92a798 2023-04-06 18:57:30,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807510715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807510716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807510825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:30,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807510825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807511031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807511036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,040 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/c1399fff78624a929548fd538fa51cac 2023-04-06 18:57:31,047 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/3742543e90914fbd84baa236e4aa6bba as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba 2023-04-06 18:57:31,051 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba, entries=150, sequenceid=599, filesize=11.9 K 2023-04-06 18:57:31,052 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/0ec7ab78f8194906ad51d459fe92a798 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798 2023-04-06 18:57:31,060 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798, entries=150, sequenceid=599, filesize=11.9 K 2023-04-06 18:57:31,061 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/c1399fff78624a929548fd538fa51cac as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac 2023-04-06 18:57:31,064 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac, entries=150, sequenceid=599, filesize=11.9 K 2023-04-06 18:57:31,065 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=152.07 KB/155720 for dd972e696d2dd9f5c10736fe54de15bd in 518ms, sequenceid=599, compaction requested=false 2023-04-06 18:57:31,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:31,065 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:31,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:31,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:31,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:31,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:31,068 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:31,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7517cd3c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:31,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:31,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,070 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:31,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:31,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:31,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,071 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:31,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:31,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:31,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:31,071 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:31,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:31,073 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:31,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:31,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:31,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:31,073 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:31,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:31,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:31,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:31,090 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:31,090 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:31,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:31,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:31,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:31,190 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:31,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:31,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:31,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:31,196 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:31,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:31,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:31,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:31,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,197 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:31,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:31,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,197 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:31,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:31,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:31,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:31,211 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:31,211 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:31,211 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:31,211 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:31,211 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=152.07 KB heapSize=446.48 KB 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:31,212 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:31,265 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/2a733b28fc1c4b06b7f9aaad29d1a5ae 2023-04-06 18:57:31,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:31,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807511367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807511368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807511452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807511454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807511456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807511477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807511487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807511684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:31,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807511691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/476543ce878e4cd28c477db42206b554 2023-04-06 18:57:31,754 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/7e33d95ab8414cf9bd94386daa476c7c 2023-04-06 18:57:31,758 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/2a733b28fc1c4b06b7f9aaad29d1a5ae as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae 2023-04-06 18:57:31,767 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae, entries=150, sequenceid=630, filesize=11.9 K 2023-04-06 18:57:31,768 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/476543ce878e4cd28c477db42206b554 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554 2023-04-06 18:57:31,772 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554, entries=150, sequenceid=630, filesize=11.9 K 2023-04-06 18:57:31,773 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/7e33d95ab8414cf9bd94386daa476c7c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c 2023-04-06 18:57:31,779 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c, entries=150, sequenceid=630, filesize=11.9 K 2023-04-06 18:57:31,780 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~446.44 KB/457152, currentSize=33.54 KB/34350 for dd972e696d2dd9f5c10736fe54de15bd in 569ms, sequenceid=630, compaction requested=true 2023-04-06 18:57:31,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:31,780 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:31,780 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:31,780 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:31,780 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,780 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:31,780 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:31,782 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,782 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,782 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,782 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:31,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36305f73[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:31,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:31,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,792 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:31,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:31,792 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:31,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:31,805 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:31,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:31,810 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:31,811 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:31,811 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:31,811 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:31,811 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:31,812 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:31,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:31,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:31,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:31,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:31,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:31,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,817 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:31,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:31,817 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:31,917 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:31,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:31,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:31,930 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:31,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:31,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:31,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:31,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:31,931 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:31,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:31,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:31,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:31,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:31,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:31,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:31,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:31,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:31,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:31,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:31,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:31,946 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:31,946 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:31,947 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:31,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:31,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807452012, max timestamp=1680807452025Num uniques 859; 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807452012, max timestamp=1680807452025Num uniques 859; 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807452012, max timestamp=1680807452025Num uniques 859; 2023-04-06 18:57:32,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1680807452042, max timestamp=1680807452059Num uniques 1018; 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46612, new segment data size=7682, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807452042, max timestamp=1680807452059Num uniques 1018; 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807452042, max timestamp=1680807452059Num uniques 1018; 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:32,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:57:32,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807452085, max timestamp=1680807452098Num uniques 1027; 2023-04-06 18:57:32,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:57:32,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807452085, max timestamp=1680807452098Num uniques 1027; 2023-04-06 18:57:32,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:57:32,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807452085, max timestamp=1680807452098Num uniques 1027; 2023-04-06 18:57:32,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 18:57:32,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807452144, max timestamp=1680807452154Num uniques 1086; 2023-04-06 18:57:32,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8515, totalHeapSize=19584, min timestamp=1680807452144, max timestamp=1680807452154Num uniques 1086; 2023-04-06 18:57:32,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49735, new segment data size=8515, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:32,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:32,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8510, totalHeapSize=19584, min timestamp=1680807452144, max timestamp=1680807452154Num uniques 1086; 2023-04-06 18:57:32,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49730, new segment data size=8510, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807452179, max timestamp=1680807452211Num uniques 1045; 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807452179, max timestamp=1680807452211Num uniques 1045; 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807452179, max timestamp=1680807452211Num uniques 1045; 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:32,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:32,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807452235, max timestamp=1680807452263Num uniques 1054; 2023-04-06 18:57:32,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 18:57:32,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807452235, max timestamp=1680807452263Num uniques 1054; 2023-04-06 18:57:32,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 18:57:32,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807452235, max timestamp=1680807452263Num uniques 1054; 2023-04-06 18:57:32,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 18:57:32,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807452294, max timestamp=1680807452303Num uniques 1063; 2023-04-06 18:57:32,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807452294, max timestamp=1680807452303Num uniques 1013; 2023-04-06 18:57:32,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-06 18:57:32,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-06 18:57:32,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807452294, max timestamp=1680807452303Num uniques 1013; 2023-04-06 18:57:32,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807452341, max timestamp=1680807452356Num uniques 1022; 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807452341, max timestamp=1680807452356Num uniques 1022; 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807452341, max timestamp=1680807452356Num uniques 1022; 2023-04-06 18:57:32,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:32,367 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/93c244be5ee94e78a0077eb4c52dfe9f 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807452372, max timestamp=1680807452401Num uniques 1031; 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807452372, max timestamp=1680807452401Num uniques 1031; 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807452372, max timestamp=1680807452401Num uniques 1031; 2023-04-06 18:57:32,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:32,405 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/4939af423b43499394228c29581db260 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807452443, max timestamp=1680807452450Num uniques 1040; 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807452443, max timestamp=1680807452450Num uniques 1040; 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807452443, max timestamp=1680807452450Num uniques 1040; 2023-04-06 18:57:32,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:57:32,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807512527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807512527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807512631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807512631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,836 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=638 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/6a0a40516b2844c792edb32838cc78e5 2023-04-06 18:57:32,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807512834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:32,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807512835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,839 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/93c244be5ee94e78a0077eb4c52dfe9f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f 2023-04-06 18:57:32,842 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f, entries=100, sequenceid=638, filesize=9.5 K 2023-04-06 18:57:32,843 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/4939af423b43499394228c29581db260 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260 2023-04-06 18:57:32,846 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260, entries=100, sequenceid=638, filesize=9.5 K 2023-04-06 18:57:32,846 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/6a0a40516b2844c792edb32838cc78e5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5 2023-04-06 18:57:32,852 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5, entries=100, sequenceid=638, filesize=9.5 K 2023-04-06 18:57:32,853 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for dd972e696d2dd9f5c10736fe54de15bd in 907ms, sequenceid=638, compaction requested=true 2023-04-06 18:57:32,854 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:32,854 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:32,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:32,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:32,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:32,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:32,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:32,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:32,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b15c66f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:32,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:32,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:32,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:32,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:32,873 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:32,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:32,873 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:32,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:32,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:32,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,875 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:32,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:32,878 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:32,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:32,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:32,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:32,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:32,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:32,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:32,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:32,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:32,879 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:32,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:32,879 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:32,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:32,979 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:32,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:32,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:32,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:32,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:32,982 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:32,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:32,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:32,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:32,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:32,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:32,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:32,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:32,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:32,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:32,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:32,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:32,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:32,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:32,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:32,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:32,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:32,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:32,998 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:32,998 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:32,999 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=147.60 KB heapSize=412.55 KB 2023-04-06 18:57:32,999 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:33,000 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:33,000 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:33,000 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:33,000 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:33,000 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:33,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:33,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:33,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807513157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807513157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807513260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807513262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,433 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/0e580550750243558800a19647a7fe0d 2023-04-06 18:57:33,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38526 deadline: 1680807513465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807513466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,472 DEBUG [Thread-1713] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:33,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38508 deadline: 1680807513470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,474 DEBUG [Thread-1711] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:33,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38620 deadline: 1680807513473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807513474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,478 DEBUG [Thread-1707] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:33,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807513771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:33,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807513779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,855 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/9c20d0ff843345d6bec2d79644d1f167 2023-04-06 18:57:33,888 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/05ab3bc956614c4d923b9fa0ce394256 2023-04-06 18:57:33,897 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/0e580550750243558800a19647a7fe0d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d 2023-04-06 18:57:33,914 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d, entries=150, sequenceid=829, filesize=11.9 K 2023-04-06 18:57:33,918 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/9c20d0ff843345d6bec2d79644d1f167 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167 2023-04-06 18:57:33,923 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167, entries=150, sequenceid=829, filesize=11.9 K 2023-04-06 18:57:33,924 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/05ab3bc956614c4d923b9fa0ce394256 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256 2023-04-06 18:57:33,932 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256, entries=150, sequenceid=829, filesize=11.9 K 2023-04-06 18:57:33,932 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~412.50 KB/422400, currentSize=40.25 KB/41220 for dd972e696d2dd9f5c10736fe54de15bd in 933ms, sequenceid=829, compaction requested=true 2023-04-06 18:57:33,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:33,932 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:33,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:33,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:33,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:33,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:33,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764feff9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:33,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:33,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,950 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:33,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:33,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:33,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:33,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:33,954 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:33,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:33,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:33,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:33,954 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:33,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:33,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:33,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:33,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:33,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:33,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:33,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:33,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:33,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:33,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:33,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:33,969 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:33,970 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:33,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:33,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:33,970 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:33,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:34,073 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:34,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:34,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:34,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:34,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:34,079 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:34,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:34,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:34,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:34,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:34,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:34,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:34,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:34,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:34,089 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:34,090 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:34,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:34,116 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/3f8ba9c50b764096a8d6d809ac39505a 2023-04-06 18:57:34,146 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/61075ac0d6734981824dc175365f0418 2023-04-06 18:57:34,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:34,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:34,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807454309, max timestamp=1680807454326Num uniques 859; 2023-04-06 18:57:34,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807454309, max timestamp=1680807454326Num uniques 859; 2023-04-06 18:57:34,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:57:34,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:57:34,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807454309, max timestamp=1680807454326Num uniques 859; 2023-04-06 18:57:34,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:57:34,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807514376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807514378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807514479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807514483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,577 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=838 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/a8dc9a9e651e40578f7d4070328a9466 2023-04-06 18:57:34,582 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/3f8ba9c50b764096a8d6d809ac39505a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a 2023-04-06 18:57:34,590 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a, entries=100, sequenceid=838, filesize=9.5 K 2023-04-06 18:57:34,591 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/61075ac0d6734981824dc175365f0418 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418 2023-04-06 18:57:34,594 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418, entries=100, sequenceid=838, filesize=9.5 K 2023-04-06 18:57:34,595 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/a8dc9a9e651e40578f7d4070328a9466 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466 2023-04-06 18:57:34,606 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466, entries=100, sequenceid=838, filesize=9.5 K 2023-04-06 18:57:34,607 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for dd972e696d2dd9f5c10736fe54de15bd in 517ms, sequenceid=838, compaction requested=true 2023-04-06 18:57:34,607 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:34,607 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:34,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:34,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,608 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,608 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,608 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:34,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7ab221[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:34,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,610 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,610 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:34,610 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:34,610 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,612 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:34,612 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:34,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:34,612 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:34,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:34,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:34,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:34,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,619 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:34,619 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:34,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:34,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:34,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:34,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:34,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:57:34,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807514701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807514704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/4738bd647c95427da908cdac0f3e2ef2 2023-04-06 18:57:34,720 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:34,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:34,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:34,738 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:34,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:34,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:34,738 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,740 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:34,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:34,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:34,746 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,746 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:34,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:34,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:34,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:34,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@905b93a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:34,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:34,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,764 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,764 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:34,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:34,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:34,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:34,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/8c76ef80b562423688b7824b66939dc6 2023-04-06 18:57:34,772 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:34,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:34,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:34,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:34,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:34,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:34,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:34,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:34,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:34,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:34,789 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:34,789 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:34,791 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:34,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:34,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:34,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807514805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:34,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807514810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,889 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:34,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:34,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:34,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:34,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:34,908 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:34,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:34,908 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:34,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:34,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:34,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:34,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:34,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:34,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:34,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:34,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:34,928 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,928 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:34,928 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:34,928 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:34,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:34,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:34,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:34,928 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:34,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:34,937 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:34,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca630cb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:34,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:34,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:34,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:34,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:34,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:34,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:34,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:34,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:34,948 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:34,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:34,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:34,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:34,948 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:34,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:34,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:34,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:34,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:34,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,980 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:34,980 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:34,980 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:34,980 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:34,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:35,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:35,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38524 deadline: 1680807515008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:35,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38568 deadline: 1680807515019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,080 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:35,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:35,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:35,084 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:35,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:35,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:35,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:35,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:35,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:35,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:35,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:35,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:35,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:35,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:35,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:35,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:35,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,109 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:35,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:35,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f2681[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:35,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:35,117 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,121 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:35,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:35,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:35,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:35,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:35,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:35,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:35,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:35,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:35,125 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:35,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:35,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:35,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:35,125 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:35,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:35,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:35,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,129 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:35,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:35,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:35,130 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:35,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:35,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:35,131 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:35,131 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:35,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:35,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:35,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:35,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/86e804ec92d5401f8bcfd990f11da1ff 2023-04-06 18:57:35,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/4738bd647c95427da908cdac0f3e2ef2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2 2023-04-06 18:57:35,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2, entries=250, sequenceid=876, filesize=16.7 K 2023-04-06 18:57:35,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/8c76ef80b562423688b7824b66939dc6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6 2023-04-06 18:57:35,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6, entries=150, sequenceid=876, filesize=11.9 K 2023-04-06 18:57:35,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/86e804ec92d5401f8bcfd990f11da1ff as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff 2023-04-06 18:57:35,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff, entries=150, sequenceid=876, filesize=11.9 K 2023-04-06 18:57:35,215 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for dd972e696d2dd9f5c10736fe54de15bd in 523ms, sequenceid=876, compaction requested=true 2023-04-06 18:57:35,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:35,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:35,215 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 18:57:35,215 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 18:57:35,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:35,215 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:35,217 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81028 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 18:57:35,217 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/B is initiating minor compaction (all files) 2023-04-06 18:57:35,217 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/B in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,217 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/898947d1e66c4ee5bf92d22a8c606c7a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=79.1 K 2023-04-06 18:57:35,218 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 898947d1e66c4ee5bf92d22a8c606c7a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:35,218 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85908 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 18:57:35,218 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0ec7ab78f8194906ad51d459fe92a798, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1680807449936 2023-04-06 18:57:35,218 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/A is initiating minor compaction (all files) 2023-04-06 18:57:35,218 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/A in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,218 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/60b48b6458024fd28cb7e74e05cfaf4e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=83.9 K 2023-04-06 18:57:35,218 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 476543ce878e4cd28c477db42206b554, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680807450603 2023-04-06 18:57:35,219 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4939af423b43499394228c29581db260, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1680807451352 2023-04-06 18:57:35,219 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 60b48b6458024fd28cb7e74e05cfaf4e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:35,219 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9c20d0ff843345d6bec2d79644d1f167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680807452517 2023-04-06 18:57:35,220 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 61075ac0d6734981824dc175365f0418, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1680807453151 2023-04-06 18:57:35,220 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 3742543e90914fbd84baa236e4aa6bba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1680807449936 2023-04-06 18:57:35,220 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8c76ef80b562423688b7824b66939dc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1680807454377 2023-04-06 18:57:35,221 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2a733b28fc1c4b06b7f9aaad29d1a5ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680807450603 2023-04-06 18:57:35,221 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 93c244be5ee94e78a0077eb4c52dfe9f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1680807451352 2023-04-06 18:57:35,222 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0e580550750243558800a19647a7fe0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680807452517 2023-04-06 18:57:35,222 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 3f8ba9c50b764096a8d6d809ac39505a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1680807453151 2023-04-06 18:57:35,222 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4738bd647c95427da908cdac0f3e2ef2, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1680807454370 2023-04-06 18:57:35,231 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:35,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:35,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:35,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:35,238 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:35,238 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:35,238 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:35,238 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:35,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:35,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:35,250 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:35,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:35,253 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#B#compaction#281 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:35,253 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#A#compaction#282 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:35,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:35,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:35,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:35,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:35,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:35,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:35,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:35,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:35,261 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,261 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. started... 2023-04-06 18:57:35,261 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:57:35,261 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:35,261 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:35,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:35,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:35,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:35,262 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:35,297 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/e4022fac4c97431e9c67cf69328a783b 2023-04-06 18:57:35,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. as already flushing 2023-04-06 18:57:35,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807455344, max timestamp=1680807455356Num uniques 859; 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807455344, max timestamp=1680807455356Num uniques 859; 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807455346, max timestamp=1680807455356Num uniques 859; 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807455404, max timestamp=1680807455415Num uniques 1068; 2023-04-06 18:57:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807455404, max timestamp=1680807455415Num uniques 1018; 2023-04-06 18:57:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:57:35,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807455404, max timestamp=1680807455415Num uniques 1068; 2023-04-06 18:57:35,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-06 18:57:35,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:57:35,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680807455437, max timestamp=1680807455461Num uniques 1077; 2023-04-06 18:57:35,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-06 18:57:35,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680807455437, max timestamp=1680807455461Num uniques 1077; 2023-04-06 18:57:35,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-06 18:57:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807455437, max timestamp=1680807455461Num uniques 1077; 2023-04-06 18:57:35,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-06 18:57:35,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807455508, max timestamp=1680807455524Num uniques 1136; 2023-04-06 18:57:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-06 18:57:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807455508, max timestamp=1680807455524Num uniques 1136; 2023-04-06 18:57:35,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-06 18:57:35,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680807455508, max timestamp=1680807455524Num uniques 1036; 2023-04-06 18:57:35,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-06 18:57:35,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807455558, max timestamp=1680807455575Num uniques 1045; 2023-04-06 18:57:35,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807455558, max timestamp=1680807455575Num uniques 1009; 2023-04-06 18:57:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680807455558, max timestamp=1680807455575Num uniques 1045; 2023-04-06 18:57:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 18:57:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-04-06 18:57:35,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807455613, max timestamp=1680807455628Num uniques 1054; 2023-04-06 18:57:35,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 18:57:35,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807455613, max timestamp=1680807455628Num uniques 1054; 2023-04-06 18:57:35,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 18:57:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9335, totalHeapSize=21456, min timestamp=1680807455613, max timestamp=1680807455628Num uniques 1054; 2023-04-06 18:57:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48265, new segment data size=9335, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 18:57:35,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/00e9d7fb12634e1ebb3f3709c0002852 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00e9d7fb12634e1ebb3f3709c0002852 2023-04-06 18:57:35,698 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/B of dd972e696d2dd9f5c10736fe54de15bd into 00e9d7fb12634e1ebb3f3709c0002852(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:35,698 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:35,698 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/B, priority=9, startTime=1680807455215; duration=0sec 2023-04-06 18:57:35,698 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:35,698 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 18:57:35,709 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/fecdebfaefc24341b87fe216d4d65ca4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/fecdebfaefc24341b87fe216d4d65ca4 2023-04-06 18:57:35,711 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/80eeb3b83119476b84ba5b0f8d1ad5c4 2023-04-06 18:57:35,713 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81028 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 18:57:35,713 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): dd972e696d2dd9f5c10736fe54de15bd/C is initiating minor compaction (all files) 2023-04-06 18:57:35,713 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dd972e696d2dd9f5c10736fe54de15bd/C in TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:35,713 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/28804447ed9e4329909b7dba8045a9cf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp, totalSize=79.1 K 2023-04-06 18:57:35,713 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 28804447ed9e4329909b7dba8045a9cf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1680807449310 2023-04-06 18:57:35,714 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c1399fff78624a929548fd538fa51cac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1680807449936 2023-04-06 18:57:35,714 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7e33d95ab8414cf9bd94386daa476c7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1680807450603 2023-04-06 18:57:35,715 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6a0a40516b2844c792edb32838cc78e5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=638, earliestPutTs=1680807451352 2023-04-06 18:57:35,715 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 05ab3bc956614c4d923b9fa0ce394256, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680807452517 2023-04-06 18:57:35,716 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a8dc9a9e651e40578f7d4070328a9466, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=838, earliestPutTs=1680807453151 2023-04-06 18:57:35,716 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 86e804ec92d5401f8bcfd990f11da1ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1680807454377 2023-04-06 18:57:35,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807455693, max timestamp=1680807455717Num uniques 1063; 2023-04-06 18:57:35,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-06 18:57:35,721 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/A of dd972e696d2dd9f5c10736fe54de15bd into fecdebfaefc24341b87fe216d4d65ca4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:35,721 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:35,721 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/A, priority=9, startTime=1680807455215; duration=0sec 2023-04-06 18:57:35,721 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:35,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807455693, max timestamp=1680807455717Num uniques 1063; 2023-04-06 18:57:35,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680807455693, max timestamp=1680807455717Num uniques 1063; 2023-04-06 18:57:35,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-06 18:57:35,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-06 18:57:35,745 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): dd972e696d2dd9f5c10736fe54de15bd#C#compaction#286 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:35,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807455761, max timestamp=1680807455771Num uniques 1022; 2023-04-06 18:57:35,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:35,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807455761, max timestamp=1680807455771Num uniques 1022; 2023-04-06 18:57:35,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:35,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680807455761, max timestamp=1680807455771Num uniques 1022; 2023-04-06 18:57:35,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807455796, max timestamp=1680807455804Num uniques 1031; 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807455796, max timestamp=1680807455804Num uniques 1031; 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807455796, max timestamp=1680807455804Num uniques 1031; 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:35,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-06 18:57:35,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807455828, max timestamp=1680807455841Num uniques 1040; 2023-04-06 18:57:35,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807455828, max timestamp=1680807455841Num uniques 1040; 2023-04-06 18:57:35,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-06 18:57:35,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-06 18:57:35,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807455828, max timestamp=1680807455841Num uniques 1040; 2023-04-06 18:57:35,846 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-06 18:57:35,887 DEBUG [Thread-1726] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1656b283 to 127.0.0.1:63695 2023-04-06 18:57:35,887 DEBUG [Thread-1726] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,888 DEBUG [Thread-1720] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13695730 to 127.0.0.1:63695 2023-04-06 18:57:35,888 DEBUG [Thread-1720] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,889 DEBUG [Thread-1715] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:63695 2023-04-06 18:57:35,889 DEBUG [Thread-1715] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,891 DEBUG [Thread-1722] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0993cc7d to 127.0.0.1:63695 2023-04-06 18:57:35,891 DEBUG [Thread-1722] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,891 DEBUG [Thread-1718] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57d25552 to 127.0.0.1:63695 2023-04-06 18:57:35,892 DEBUG [Thread-1709] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43184c57 to 127.0.0.1:63695 2023-04-06 18:57:35,892 DEBUG [Thread-1718] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,892 DEBUG [Thread-1709] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:35,894 DEBUG [Thread-1724] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69b9b3fd to 127.0.0.1:63695 2023-04-06 18:57:35,894 DEBUG [Thread-1724] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:36,167 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=883 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/047a21b896084fe39b7ec021f82b7148 2023-04-06 18:57:36,172 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/e4022fac4c97431e9c67cf69328a783b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/e4022fac4c97431e9c67cf69328a783b 2023-04-06 18:57:36,176 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/e4022fac4c97431e9c67cf69328a783b, entries=50, sequenceid=883, filesize=7.1 K 2023-04-06 18:57:36,177 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/80eeb3b83119476b84ba5b0f8d1ad5c4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/80eeb3b83119476b84ba5b0f8d1ad5c4 2023-04-06 18:57:36,185 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/80eeb3b83119476b84ba5b0f8d1ad5c4, entries=50, sequenceid=883, filesize=7.1 K 2023-04-06 18:57:36,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/047a21b896084fe39b7ec021f82b7148 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/047a21b896084fe39b7ec021f82b7148 2023-04-06 18:57:36,189 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/047a21b896084fe39b7ec021f82b7148, entries=50, sequenceid=883, filesize=7.1 K 2023-04-06 18:57:36,190 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/3c23253b47004ecebcd3848fa282707d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/3c23253b47004ecebcd3848fa282707d 2023-04-06 18:57:36,190 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=120.76 KB/123660 for dd972e696d2dd9f5c10736fe54de15bd in 929ms, sequenceid=883, compaction requested=false 2023-04-06 18:57:36,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:36,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:36,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:36,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:36,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:36,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d9c77a9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:36,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:36,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:36,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:36,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:36,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:36,195 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in dd972e696d2dd9f5c10736fe54de15bd/C of dd972e696d2dd9f5c10736fe54de15bd into 3c23253b47004ecebcd3848fa282707d(size=12.7 K), total size for store is 19.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:36,195 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:36,195 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd., storeName=dd972e696d2dd9f5c10736fe54de15bd/C, priority=9, startTime=1680807455215; duration=0sec 2023-04-06 18:57:36,195 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:36,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:36,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:36,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:36,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:36,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:36,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:36,214 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:36,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:36,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:36,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:36,214 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:36,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:36,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:36,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:36,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:36,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:36,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:36,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:36,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:36,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:36,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:36,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:36,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:36,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:36,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:36,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:36,218 DEBUG [Thread-1717] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:36,219 DEBUG [Thread-1717] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:36,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:36,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:36,320 DEBUG [Thread-1717] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:36,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:37,480 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=127.47 KB heapSize=362.16 KB 2023-04-06 18:57:37,480 DEBUG [Thread-1713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:63695 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:37,480 DEBUG [Thread-1713] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:37,480 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:37,485 DEBUG [Thread-1707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29a548ac to 127.0.0.1:63695 2023-04-06 18:57:37,485 DEBUG [Thread-1707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:37,494 DEBUG [Thread-1711] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3efec6 to 127.0.0.1:63695 2023-04-06 18:57:37,494 DEBUG [Thread-1711] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:37,494 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:57:37,494 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-06 18:57:37,494 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 314 2023-04-06 18:57:37,494 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 308 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4687 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4748 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4794 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4710 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 4693 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:57:37,495 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:57:37,495 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12394d27 to 127.0.0.1:63695 2023-04-06 18:57:37,495 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:57:37,496 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:57:37,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:57:37,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:37,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:37,509 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807457509"}]},"ts":"1680807457509"} 2023-04-06 18:57:37,511 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:57:37,514 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:57:37,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, UNASSIGN}] 2023-04-06 18:57:37,515 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, UNASSIGN 2023-04-06 18:57:37,516 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=dd972e696d2dd9f5c10736fe54de15bd, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:37,516 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807457516"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807457516"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807457516"}]},"ts":"1680807457516"} 2023-04-06 18:57:37,517 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:37,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:37,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing dd972e696d2dd9f5c10736fe54de15bd, disabling compactions & flushes 2023-04-06 18:57:37,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:37,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:37,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1076 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/6d0bf977539344d1aa96ffa956cd0fc1 2023-04-06 18:57:38,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:38,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1076 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/81d5c987d92243cd84de29d25f0a98bb 2023-04-06 18:57:38,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:38,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1076 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/2b0961eb97634580a995edca4596621c 2023-04-06 18:57:38,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/6d0bf977539344d1aa96ffa956cd0fc1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6d0bf977539344d1aa96ffa956cd0fc1 2023-04-06 18:57:38,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6d0bf977539344d1aa96ffa956cd0fc1, entries=150, sequenceid=1076, filesize=11.9 K 2023-04-06 18:57:38,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/81d5c987d92243cd84de29d25f0a98bb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/81d5c987d92243cd84de29d25f0a98bb 2023-04-06 18:57:38,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/81d5c987d92243cd84de29d25f0a98bb, entries=150, sequenceid=1076, filesize=11.9 K 2023-04-06 18:57:38,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/2b0961eb97634580a995edca4596621c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2b0961eb97634580a995edca4596621c 2023-04-06 18:57:38,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2b0961eb97634580a995edca4596621c, entries=150, sequenceid=1076, filesize=11.9 K 2023-04-06 18:57:38,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~362.11 KB/370800, currentSize=13.42 KB/13740 for dd972e696d2dd9f5c10736fe54de15bd in 1264ms, sequenceid=1076, compaction requested=true 2023-04-06 18:57:38,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:38,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:38,744 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. because compaction request was cancelled 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. after waiting 0 ms 2023-04-06 18:57:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:38,744 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. because compaction request was cancelled 2023-04-06 18:57:38,744 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. because compaction request was cancelled 2023-04-06 18:57:38,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing dd972e696d2dd9f5c10736fe54de15bd 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-06 18:57:38,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=A 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=B 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dd972e696d2dd9f5c10736fe54de15bd, store=C 2023-04-06 18:57:38,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:39,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/c253699d95e04cb5852ae2215dda6f4a 2023-04-06 18:57:39,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/bb384f64d38741b082ba6b5f52575897 2023-04-06 18:57:39,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/236b8f8fb31047d493fd856fd08d4bd7 2023-04-06 18:57:39,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/A/c253699d95e04cb5852ae2215dda6f4a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c253699d95e04cb5852ae2215dda6f4a 2023-04-06 18:57:39,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c253699d95e04cb5852ae2215dda6f4a, entries=50, sequenceid=1081, filesize=7.1 K 2023-04-06 18:57:39,190 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/B/bb384f64d38741b082ba6b5f52575897 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/bb384f64d38741b082ba6b5f52575897 2023-04-06 18:57:39,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/bb384f64d38741b082ba6b5f52575897, entries=50, sequenceid=1081, filesize=7.1 K 2023-04-06 18:57:39,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/.tmp/C/236b8f8fb31047d493fd856fd08d4bd7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/236b8f8fb31047d493fd856fd08d4bd7 2023-04-06 18:57:39,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/236b8f8fb31047d493fd856fd08d4bd7, entries=50, sequenceid=1081, filesize=7.1 K 2023-04-06 18:57:39,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for dd972e696d2dd9f5c10736fe54de15bd in 454ms, sequenceid=1081, compaction requested=true 2023-04-06 18:57:39,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/cbac54ddd4a949d896c37d250e779b63, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/540b7f1ca96f47b686d1f146a52971d0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/36e02eb297f2457189da148235089f52, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/60b48b6458024fd28cb7e74e05cfaf4e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2] to archive 2023-04-06 18:57:39,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:39,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0febdde148ef43c59f4a5efb992f6a83 2023-04-06 18:57:39,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7be97a63ddf04f99bca3c461b9f0c794 2023-04-06 18:57:39,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/af75cb7dffdd4ec6ab3ba0c9377b626d 2023-04-06 18:57:39,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5c0cd848555040d0b73696a62c223bd2 2023-04-06 18:57:39,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7ec6b6ed164947a590a48df2accb05de 2023-04-06 18:57:39,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/5f70392413e649f69fbe5acfada98208 2023-04-06 18:57:39,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9697c63756f84f5dbe12d157c2390a16 2023-04-06 18:57:39,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/9bd97e0000494625823c199fde76f88e 2023-04-06 18:57:39,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/cbac54ddd4a949d896c37d250e779b63 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/cbac54ddd4a949d896c37d250e779b63 2023-04-06 18:57:39,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/b79463049e584a76a50310ccbb2e37e2 2023-04-06 18:57:39,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/aa757276b31c42b79329d74304fc3509 2023-04-06 18:57:39,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/39f8e41858644f9fa6e6258808028575 2023-04-06 18:57:39,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/540b7f1ca96f47b686d1f146a52971d0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/540b7f1ca96f47b686d1f146a52971d0 2023-04-06 18:57:39,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/7a0409c647a2410f88b4850e7f874cf4 2023-04-06 18:57:39,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/77895d0ddf364fd0bef1065a0871281c 2023-04-06 18:57:39,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/36e02eb297f2457189da148235089f52 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/36e02eb297f2457189da148235089f52 2023-04-06 18:57:39,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c36b0b957bb1412fb02f7074d28e4f96 2023-04-06 18:57:39,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/60b48b6458024fd28cb7e74e05cfaf4e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/60b48b6458024fd28cb7e74e05cfaf4e 2023-04-06 18:57:39,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6b9e6b06bc324d188928ab672fb505d4 2023-04-06 18:57:39,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3742543e90914fbd84baa236e4aa6bba 2023-04-06 18:57:39,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/2a733b28fc1c4b06b7f9aaad29d1a5ae 2023-04-06 18:57:39,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/93c244be5ee94e78a0077eb4c52dfe9f 2023-04-06 18:57:39,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/0e580550750243558800a19647a7fe0d 2023-04-06 18:57:39,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/3f8ba9c50b764096a8d6d809ac39505a 2023-04-06 18:57:39,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/4738bd647c95427da908cdac0f3e2ef2 2023-04-06 18:57:39,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/20662b67d8274c7084705d4e13b3c643, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9792fea5d6054eb0a1c457b26edfbde4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/48518a6b3d8049a0b3e8dadd34d6439e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/898947d1e66c4ee5bf92d22a8c606c7a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6] to archive 2023-04-06 18:57:39,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:39,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/a458b31a391d4760a9efac18b67c0689 2023-04-06 18:57:39,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/b58aadbc8d54453c9425aec2ece78e3f 2023-04-06 18:57:39,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/60112e7b87244946ba5f8db94544cb33 2023-04-06 18:57:39,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/c5427262503e43b0969caa486a7f0005 2023-04-06 18:57:39,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/2f633896275345ebb282898930451be8 2023-04-06 18:57:39,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d389ddf21c804f4990987aeba0a29d62 2023-04-06 18:57:39,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/d9ed5af582fe47d0aeea14044ea9adc4 2023-04-06 18:57:39,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/20662b67d8274c7084705d4e13b3c643 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/20662b67d8274c7084705d4e13b3c643 2023-04-06 18:57:39,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/aae71273857e4076b812b70ab31a3c6c 2023-04-06 18:57:39,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/178977bb96844dc1b1e1c1f8774549f8 2023-04-06 18:57:39,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/84d893ad800640f98e3f3fd191dce0fe 2023-04-06 18:57:39,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9792fea5d6054eb0a1c457b26edfbde4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9792fea5d6054eb0a1c457b26edfbde4 2023-04-06 18:57:39,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/62e6539f3e4e46bb8674b86f9ebfb771 2023-04-06 18:57:39,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/544f567b5dd747229d6dce7e988bdf1f 2023-04-06 18:57:39,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/48518a6b3d8049a0b3e8dadd34d6439e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/48518a6b3d8049a0b3e8dadd34d6439e 2023-04-06 18:57:39,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00ed24f1993b406e8f4ded06a0bdba07 2023-04-06 18:57:39,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/e36d7a127fea4405b4c1f6d13a72304e 2023-04-06 18:57:39,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/898947d1e66c4ee5bf92d22a8c606c7a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/898947d1e66c4ee5bf92d22a8c606c7a 2023-04-06 18:57:39,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/6526c9fb1f734f229d1460e738da3e45 2023-04-06 18:57:39,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/0ec7ab78f8194906ad51d459fe92a798 2023-04-06 18:57:39,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/476543ce878e4cd28c477db42206b554 2023-04-06 18:57:39,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/4939af423b43499394228c29581db260 2023-04-06 18:57:39,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/9c20d0ff843345d6bec2d79644d1f167 2023-04-06 18:57:39,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/61075ac0d6734981824dc175365f0418 2023-04-06 18:57:39,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/8c76ef80b562423688b7824b66939dc6 2023-04-06 18:57:39,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/750fb7eef440448b9443bbe3f2ce2241, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/f05ce3bd1d4a4df598c5350fc61e6f99, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b903c60aa3664b54a5b6e864127380d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/28804447ed9e4329909b7dba8045a9cf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff] to archive 2023-04-06 18:57:39,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:57:39,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/46624752d2f24c54a21c45f52e6456c1 2023-04-06 18:57:39,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c5bb21aa3d2346438de37e37f42d98f6 2023-04-06 18:57:39,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/106f2c8671b94f90b1fcfd6c0fd6e74f 2023-04-06 18:57:39,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/211cf4a7890a4a38a9edad33f5ef8b47 2023-04-06 18:57:39,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/407ec92bbf6441a4ae53ef14c53c1b49 2023-04-06 18:57:39,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2653a66b85d847bc828f845471eea1d7 2023-04-06 18:57:39,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/acad90bfcdf849daac5a6a2fd3702704 2023-04-06 18:57:39,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/750fb7eef440448b9443bbe3f2ce2241 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/750fb7eef440448b9443bbe3f2ce2241 2023-04-06 18:57:39,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b1d7dd1660c6446ea28752941b081645 2023-04-06 18:57:39,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/8e0e12d40c214a2ebbe760192e02844f 2023-04-06 18:57:39,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/d8ff2625cc8149f7ac640dc949c40c9c 2023-04-06 18:57:39,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/f05ce3bd1d4a4df598c5350fc61e6f99 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/f05ce3bd1d4a4df598c5350fc61e6f99 2023-04-06 18:57:39,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b3921224d5f44e0bbc68fdbbbb180197 2023-04-06 18:57:39,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/e14589b68db54a52af31633156be99f9 2023-04-06 18:57:39,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b903c60aa3664b54a5b6e864127380d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/b903c60aa3664b54a5b6e864127380d7 2023-04-06 18:57:39,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/4e1e6ee05f0549ef86dd2e70d5658369 2023-04-06 18:57:39,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/9f5eae5df207409b84dd3400b6152433 2023-04-06 18:57:39,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/28804447ed9e4329909b7dba8045a9cf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/28804447ed9e4329909b7dba8045a9cf 2023-04-06 18:57:39,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/163a67d622a04a7ca414e9731e80545b 2023-04-06 18:57:39,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/c1399fff78624a929548fd538fa51cac 2023-04-06 18:57:39,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/7e33d95ab8414cf9bd94386daa476c7c 2023-04-06 18:57:39,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/6a0a40516b2844c792edb32838cc78e5 2023-04-06 18:57:39,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/05ab3bc956614c4d923b9fa0ce394256 2023-04-06 18:57:39,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/a8dc9a9e651e40578f7d4070328a9466 2023-04-06 18:57:39,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/86e804ec92d5401f8bcfd990f11da1ff 2023-04-06 18:57:39,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/recovered.edits/1084.seqid, newMaxSeqId=1084, maxSeqId=1 2023-04-06 18:57:39,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd. 2023-04-06 18:57:39,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for dd972e696d2dd9f5c10736fe54de15bd: 2023-04-06 18:57:39,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:39,324 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=dd972e696d2dd9f5c10736fe54de15bd, regionState=CLOSED 2023-04-06 18:57:39,324 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807459324"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807459324"}]},"ts":"1680807459324"} 2023-04-06 18:57:39,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-06 18:57:39,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure dd972e696d2dd9f5c10736fe54de15bd, server=jenkins-hbase20.apache.org,34607,1680807373848 in 1.8080 sec 2023-04-06 18:57:39,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-06 18:57:39,328 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=dd972e696d2dd9f5c10736fe54de15bd, UNASSIGN in 1.8110 sec 2023-04-06 18:57:39,328 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807459328"}]},"ts":"1680807459328"} 2023-04-06 18:57:39,329 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:57:39,330 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:57:39,331 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8340 sec 2023-04-06 18:57:39,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 18:57:39,613 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-06 18:57:39,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:57:39,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,618 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 18:57:39,626 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:39,629 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/recovered.edits] 2023-04-06 18:57:39,633 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6d0bf977539344d1aa96ffa956cd0fc1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/6d0bf977539344d1aa96ffa956cd0fc1 2023-04-06 18:57:39,635 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c253699d95e04cb5852ae2215dda6f4a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/c253699d95e04cb5852ae2215dda6f4a 2023-04-06 18:57:39,637 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/e4022fac4c97431e9c67cf69328a783b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/e4022fac4c97431e9c67cf69328a783b 2023-04-06 18:57:39,639 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/fecdebfaefc24341b87fe216d4d65ca4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/A/fecdebfaefc24341b87fe216d4d65ca4 2023-04-06 18:57:39,642 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00e9d7fb12634e1ebb3f3709c0002852 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/00e9d7fb12634e1ebb3f3709c0002852 2023-04-06 18:57:39,643 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/80eeb3b83119476b84ba5b0f8d1ad5c4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/80eeb3b83119476b84ba5b0f8d1ad5c4 2023-04-06 18:57:39,645 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/81d5c987d92243cd84de29d25f0a98bb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/81d5c987d92243cd84de29d25f0a98bb 2023-04-06 18:57:39,647 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/bb384f64d38741b082ba6b5f52575897 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/B/bb384f64d38741b082ba6b5f52575897 2023-04-06 18:57:39,651 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/047a21b896084fe39b7ec021f82b7148 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/047a21b896084fe39b7ec021f82b7148 2023-04-06 18:57:39,653 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/236b8f8fb31047d493fd856fd08d4bd7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/236b8f8fb31047d493fd856fd08d4bd7 2023-04-06 18:57:39,654 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2b0961eb97634580a995edca4596621c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/2b0961eb97634580a995edca4596621c 2023-04-06 18:57:39,655 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/3c23253b47004ecebcd3848fa282707d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/C/3c23253b47004ecebcd3848fa282707d 2023-04-06 18:57:39,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/recovered.edits/1084.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd/recovered.edits/1084.seqid 2023-04-06 18:57:39,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/dd972e696d2dd9f5c10736fe54de15bd 2023-04-06 18:57:39,657 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:57:39,659 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,660 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:57:39,662 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:57:39,663 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,663 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:57:39,663 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807459663"}]},"ts":"9223372036854775807"} 2023-04-06 18:57:39,665 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:57:39,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => dd972e696d2dd9f5c10736fe54de15bd, NAME => 'TestAcidGuarantees,,1680807433673.dd972e696d2dd9f5c10736fe54de15bd.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:57:39,665 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:57:39,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807459665"}]},"ts":"9223372036854775807"} 2023-04-06 18:57:39,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:57:39,667 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,668 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-04-06 18:57:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 18:57:39,720 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-06 18:57:39,732 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 231), OpenFileDescriptor=554 (was 548) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=527 (was 400) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=2855 (was 2912) 2023-04-06 18:57:39,746 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=554, MaxFileDescriptor=60000, SystemLoadAverage=527, ProcessCount=166, AvailableMemoryMB=2854 2023-04-06 18:57:39,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:57:39,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:57:39,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:39,750 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:57:39,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-06 18:57:39,751 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:57:39,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:39,752 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:39,752 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 empty. 2023-04-06 18:57:39,753 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:39,753 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:57:39,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:40,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:40,160 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:57:40,161 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 08d5d1fdb4f1757f6fcf6168ecd0a421, NAME => 'TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:57:40,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:40,568 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:57:40,568 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 08d5d1fdb4f1757f6fcf6168ecd0a421, disabling compactions & flushes 2023-04-06 18:57:40,568 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,568 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,568 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. after waiting 0 ms 2023-04-06 18:57:40,568 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,568 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,569 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:40,571 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:57:40,572 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807460572"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807460572"}]},"ts":"1680807460572"} 2023-04-06 18:57:40,573 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:57:40,574 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:57:40,574 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807460574"}]},"ts":"1680807460574"} 2023-04-06 18:57:40,575 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:57:40,578 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, ASSIGN}] 2023-04-06 18:57:40,580 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, ASSIGN 2023-04-06 18:57:40,580 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:57:40,732 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:40,733 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807460732"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807460732"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807460732"}]},"ts":"1680807460732"} 2023-04-06 18:57:40,735 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:40,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:40,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08d5d1fdb4f1757f6fcf6168ecd0a421, NAME => 'TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:57:40,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:57:40,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,901 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,903 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A 2023-04-06 18:57:40,903 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A 2023-04-06 18:57:40,903 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:40,903 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName A 2023-04-06 18:57:40,904 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:40,904 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,905 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B 2023-04-06 18:57:40,905 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B 2023-04-06 18:57:40,905 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:40,906 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName B 2023-04-06 18:57:40,906 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:40,906 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,908 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C 2023-04-06 18:57:40,908 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C 2023-04-06 18:57:40,908 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:40,908 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName C 2023-04-06 18:57:40,909 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:40,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:57:40,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:40,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:57:40,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08d5d1fdb4f1757f6fcf6168ecd0a421; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72307294, jitterRate=0.07746264338493347}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:57:40,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:40,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., pid=37, masterSystemTime=1680807460889 2023-04-06 18:57:40,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:40,919 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:40,919 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807460919"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807460919"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807460919"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807460919"}]},"ts":"1680807460919"} 2023-04-06 18:57:40,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-06 18:57:40,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 in 185 msec 2023-04-06 18:57:40,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-06 18:57:40,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, ASSIGN in 344 msec 2023-04-06 18:57:40,924 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:57:40,924 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807460924"}]},"ts":"1680807460924"} 2023-04-06 18:57:40,925 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:57:40,926 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:57:40,928 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1780 sec 2023-04-06 18:57:41,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 18:57:41,862 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-06 18:57:41,863 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x74a355de to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:41,872 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a98302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:41,873 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:41,875 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:41,876 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:57:41,877 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:57:41,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:57:41,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:57:41,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 18:57:42,293 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:57:42,296 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:57:42,296 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:57:42,300 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 18:57:42,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, REOPEN/MOVE}] 2023-04-06 18:57:42,305 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, REOPEN/MOVE 2023-04-06 18:57:42,306 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,306 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807462306"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807462306"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807462306"}]},"ts":"1680807462306"} 2023-04-06 18:57:42,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:42,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08d5d1fdb4f1757f6fcf6168ecd0a421, disabling compactions & flushes 2023-04-06 18:57:42,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. after waiting 0 ms 2023-04-06 18:57:42,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 18:57:42,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:42,471 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 08d5d1fdb4f1757f6fcf6168ecd0a421 to self. 2023-04-06 18:57:42,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,474 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=CLOSED 2023-04-06 18:57:42,474 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807462474"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807462474"}]},"ts":"1680807462474"} 2023-04-06 18:57:42,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-06 18:57:42,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 in 168 msec 2023-04-06 18:57:42,477 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=true 2023-04-06 18:57:42,627 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807462627"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807462627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807462627"}]},"ts":"1680807462627"} 2023-04-06 18:57:42,630 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:57:42,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 08d5d1fdb4f1757f6fcf6168ecd0a421, NAME => 'TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:57:42,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:57:42,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,792 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,794 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A 2023-04-06 18:57:42,794 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A 2023-04-06 18:57:42,794 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:42,795 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName A 2023-04-06 18:57:42,797 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:42,798 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,799 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B 2023-04-06 18:57:42,799 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B 2023-04-06 18:57:42,799 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:42,800 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName B 2023-04-06 18:57:42,800 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:42,801 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,802 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C 2023-04-06 18:57:42,802 DEBUG [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C 2023-04-06 18:57:42,802 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:57:42,803 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 08d5d1fdb4f1757f6fcf6168ecd0a421 columnFamilyName C 2023-04-06 18:57:42,803 INFO [StoreOpener-08d5d1fdb4f1757f6fcf6168ecd0a421-1] regionserver.HStore(310): Store=08d5d1fdb4f1757f6fcf6168ecd0a421/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:57:42,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:57:42,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 08d5d1fdb4f1757f6fcf6168ecd0a421; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64736426, jitterRate=-0.035352081060409546}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:57:42,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:42,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., pid=42, masterSystemTime=1680807462782 2023-04-06 18:57:42,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,816 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,817 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807462816"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807462816"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807462816"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807462816"}]},"ts":"1680807462816"} 2023-04-06 18:57:42,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-06 18:57:42,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 in 188 msec 2023-04-06 18:57:42,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-06 18:57:42,824 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, REOPEN/MOVE in 517 msec 2023-04-06 18:57:42,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-06 18:57:42,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 525 msec 2023-04-06 18:57:42,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 948 msec 2023-04-06 18:57:42,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-06 18:57:42,833 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x57b6628e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,840 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39d5c003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,840 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,847 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,847 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,853 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,854 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,859 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,860 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x52249391 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,866 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a9ece3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,866 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x2ded66ad to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,872 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,873 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x14d061d2 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,877 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52cea5eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,878 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x228ff1a4 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,885 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3546dad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,885 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x143a1290 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,889 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4965c362, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,889 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5e613a44 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:57:42,894 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a0b0d8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:57:42,899 DEBUG [hconnection-0x17ff5837-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,900 DEBUG [hconnection-0x495b9b5d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,900 DEBUG [hconnection-0x6add172b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:42,902 DEBUG [hconnection-0x734075c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,902 DEBUG [hconnection-0x4754a5d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,903 DEBUG [hconnection-0x63a77640-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,902 DEBUG [hconnection-0x2573df4b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,902 DEBUG [hconnection-0x4cd14457-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,902 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,903 DEBUG [hconnection-0x14a687d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,902 DEBUG [hconnection-0x13d67e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:57:42,905 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42990, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,903 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:42,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:42,910 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,911 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:42,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:42,911 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43004, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:42,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,912 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,912 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,914 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:42,916 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:57:42,917 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:42,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:42,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:42,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:42,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:42,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:42,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:42,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:42,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:42,935 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:57:42,936 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,936 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:42,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:42,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:42,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:42,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:42,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:42,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:42,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:42,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:42,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f4cf49[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:42,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:42,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:42,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:42,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:42,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:42,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:42,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:42,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:42,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:42,946 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:42,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:42,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:42,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:42,947 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:42,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:42,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:42,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:42,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:42,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:42,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:42,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:42,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:42,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:42,955 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:42,955 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:42,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:42,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807522953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807522954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:42,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:42,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:42,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807522954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807522956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:42,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:42,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807522956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,055 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807523056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,060 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807523056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807523057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,061 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,061 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807523057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807523057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,061 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,061 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,065 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,066 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,066 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,067 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,067 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e98ed4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,070 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,072 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,072 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,072 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,075 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,075 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:43,175 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,179 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,179 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,180 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,181 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,181 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,181 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,181 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,181 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,181 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,181 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,182 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,182 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,182 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,182 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,183 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,183 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,183 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8be05e6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,185 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,186 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,187 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,187 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,188 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,188 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,188 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,189 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,190 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,191 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,191 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807523261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807523262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807523262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807523263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807523264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,292 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,294 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,296 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,298 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6910a67[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,299 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,299 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,300 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,300 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,300 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,302 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,303 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,304 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,304 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,304 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,304 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,308 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,308 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,308 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:43,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,370 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406881c241df6c5413ab299e1bf63d1e109 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406881c241df6c5413ab299e1bf63d1e109 2023-04-06 18:57:43,408 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,410 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,411 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,411 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,412 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,412 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,413 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,413 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,414 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,415 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f0137b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,415 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,416 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,417 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,417 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,417 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,419 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,419 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,420 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,422 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,423 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,423 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:43,523 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,526 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,528 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,528 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,528 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,528 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,529 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,529 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,531 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5157afbf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,531 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,531 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,531 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,531 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,531 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,531 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,531 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,532 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,532 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,532 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,532 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,535 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,535 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,535 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,535 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,535 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,535 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,538 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,538 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,538 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,538 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:43,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,539 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,539 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,539 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807523565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807523565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807523565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807523566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807523569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,639 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,641 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,641 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,646 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29be0146[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,647 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,647 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,648 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,650 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,650 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,651 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,654 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,655 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,655 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,755 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,757 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,757 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,757 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,757 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,758 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:43,761 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:43,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:43,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:43,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,762 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:43,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2772f424[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:43,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:43,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,765 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:43,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:43,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:43,765 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:43,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:43,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,766 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:43,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:43,769 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:43,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:43,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:43,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:43,769 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:43,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:43,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:43,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,771 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:43,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:43,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:43,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,774 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:43,774 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:43,774 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/f75edc0776b442ff811f68cb71f03424 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:43,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:43,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/384b9753b3bc4077af8f594c24a58b0f 2023-04-06 18:57:43,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/2a9d59497c1f4f169fb5e75a68fa278f 2023-04-06 18:57:43,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/f75edc0776b442ff811f68cb71f03424 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424 2023-04-06 18:57:43,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424, entries=250, sequenceid=18, filesize=38.7 K 2023-04-06 18:57:43,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/384b9753b3bc4077af8f594c24a58b0f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f 2023-04-06 18:57:43,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f, entries=150, sequenceid=18, filesize=11.6 K 2023-04-06 18:57:43,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/2a9d59497c1f4f169fb5e75a68fa278f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f 2023-04-06 18:57:43,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f, entries=150, sequenceid=18, filesize=11.6 K 2023-04-06 18:57:43,830 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 895ms, sequenceid=18, compaction requested=false 2023-04-06 18:57:43,831 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 18:57:43,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:43,874 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:43,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:43,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:43,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:43,876 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:43,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:43,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:43,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,880 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:43,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:43,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:43,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:43,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:43,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:43,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:43,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:43,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:43,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:43,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:43,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:43,881 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:43,881 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:43,881 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:43,882 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:44,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:44,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807524096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807524096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807524124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807524124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807524124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807524225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807524225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807524234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807524234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807524234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,292 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406615f52b7ebe640e6a7db59a9e71370d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406615f52b7ebe640e6a7db59a9e71370d7 2023-04-06 18:57:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807524429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807524429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807524437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807524438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807524438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,708 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/0a8c7c86cad3456ab3f010d8e0c48dbd 2023-04-06 18:57:44,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807524733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807524733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807524742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807524743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:44,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807524744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:44,846 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:57:45,117 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/05894bce47e64a5ba24daf8bbab59f35 2023-04-06 18:57:45,124 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/92bddfdd8a9a4dec9576e0750ab7638e 2023-04-06 18:57:45,127 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/0a8c7c86cad3456ab3f010d8e0c48dbd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd 2023-04-06 18:57:45,129 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd, entries=150, sequenceid=37, filesize=25.1 K 2023-04-06 18:57:45,130 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/05894bce47e64a5ba24daf8bbab59f35 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35 2023-04-06 18:57:45,133 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35, entries=150, sequenceid=37, filesize=11.6 K 2023-04-06 18:57:45,133 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/92bddfdd8a9a4dec9576e0750ab7638e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e 2023-04-06 18:57:45,136 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e, entries=150, sequenceid=37, filesize=11.6 K 2023-04-06 18:57:45,137 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1256ms, sequenceid=37, compaction requested=false 2023-04-06 18:57:45,137 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:45,137 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,137 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef0507a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,140 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,140 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,140 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,143 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,143 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,144 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,144 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,148 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,148 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:45,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:45,239 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:45,248 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,249 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b6efd7b1706244e2aee6365374d40985 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6efd7b1706244e2aee6365374d40985 2023-04-06 18:57:45,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,252 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,253 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,254 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,254 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,255 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,255 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,255 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,255 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,256 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,256 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2236344c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,258 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,260 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,261 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,261 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,261 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,261 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,264 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,264 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,264 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,264 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,265 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,265 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807525257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807525257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807525258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807525260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807525261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,365 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,369 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807525366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807525366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807525367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807525367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807525367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b5bb57d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,375 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,377 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,377 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,379 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,381 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,382 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,382 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,482 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,484 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,484 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,487 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,487 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,487 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,487 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,487 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,488 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b38ade6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,490 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,490 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,491 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,493 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,493 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,495 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,497 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,497 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807525571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807525571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807525571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807525572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807525572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,597 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,600 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,603 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,603 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,603 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,603 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,603 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,604 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,604 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,605 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@707577b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,606 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,607 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,609 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,609 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,611 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,611 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,611 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,613 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,614 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,615 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,615 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,655 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/58e963fa17cb44e9a865cb4d4d520524 2023-04-06 18:57:45,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/856b6bbe4719435b84b20e10bae409da 2023-04-06 18:57:45,715 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,718 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,720 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,720 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14009783[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,725 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,725 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,725 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,728 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,728 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,729 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,731 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,731 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,832 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,834 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,836 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,836 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d62823[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,841 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,843 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,843 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,848 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,848 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807525874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807525875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807525876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807525878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807525878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,948 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:45,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:45,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:45,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:45,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:45,951 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:45,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:45,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:45,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:45,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:45,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:45,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:45,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:45,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:45,954 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:45,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:45,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c0ff402[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:45,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:45,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:45,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:45,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:45,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:45,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:45,958 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:45,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:45,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:45,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:45,958 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:45,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:45,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:45,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:45,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:45,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:45,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:45,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:45,963 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:45,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:45,964 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:45,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:45,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,064 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:46,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:46,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:46,066 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,067 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:46,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:46,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:46,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:46,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:46,069 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,069 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:46,069 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:46,069 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:46,069 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:46,070 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:46,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:46,071 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e434c2ec893940bf9330b733d7445925 2023-04-06 18:57:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:46,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:46,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ec077[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:46,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:46,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:46,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/58e963fa17cb44e9a865cb4d4d520524 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524 2023-04-06 18:57:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:46,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:46,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:46,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:46,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524, entries=200, sequenceid=52, filesize=31.9 K 2023-04-06 18:57:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/856b6bbe4719435b84b20e10bae409da as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da 2023-04-06 18:57:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:46,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:46,078 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:46,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:46,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:46,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:46,078 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:46,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:46,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:46,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 18:57:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:46,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e434c2ec893940bf9330b733d7445925 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925 2023-04-06 18:57:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:46,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:46,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:46,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:46,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,083 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:46,083 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:46,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:46,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:46,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 18:57:46,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 847ms, sequenceid=52, compaction requested=true 2023-04-06 18:57:46,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:57:46,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:46,087 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:46,088 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,088 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,088 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=95.8 K 2023-04-06 18:57:46,088 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=34.8 K 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f75edc0776b442ff811f68cb71f03424, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680807462927 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2a9d59497c1f4f169fb5e75a68fa278f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680807462927 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0a8c7c86cad3456ab3f010d8e0c48dbd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680807462945 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 92bddfdd8a9a4dec9576e0750ab7638e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680807462945 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 58e963fa17cb44e9a865cb4d4d520524, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464095 2023-04-06 18:57:46,088 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e434c2ec893940bf9330b733d7445925, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464104 2023-04-06 18:57:46,094 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#302 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:46,110 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#303 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:46,112 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/8a426fc4f74444f19afd34a220090da2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a426fc4f74444f19afd34a220090da2 2023-04-06 18:57:46,131 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 8a426fc4f74444f19afd34a220090da2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:46,131 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:46,131 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807466086; duration=0sec 2023-04-06 18:57:46,131 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:46,131 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:46,133 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:46,133 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:46,133 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,133 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=34.8 K 2023-04-06 18:57:46,133 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 384b9753b3bc4077af8f594c24a58b0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680807462927 2023-04-06 18:57:46,134 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 05894bce47e64a5ba24daf8bbab59f35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680807462945 2023-04-06 18:57:46,134 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 856b6bbe4719435b84b20e10bae409da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464104 2023-04-06 18:57:46,138 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:46,183 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:46,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:46,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:46,186 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:46,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:46,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:46,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:46,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:46,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:46,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:46,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:46,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:46,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:46,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:46,188 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,189 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:46,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:46,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:46,189 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:46,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:46,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:46,390 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807526401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807526401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807526402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807526403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807526403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807526508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807526508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807526508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807526509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807526509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,547 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/f32de11d8ee24e4497bbc8483a98a33f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f32de11d8ee24e4497bbc8483a98a33f 2023-04-06 18:57:46,552 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into f32de11d8ee24e4497bbc8483a98a33f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:46,552 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:46,552 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807466086; duration=0sec 2023-04-06 18:57:46,552 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:46,605 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066ef798fac08d4f1c92509875a8bc3f35 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066ef798fac08d4f1c92509875a8bc3f35 2023-04-06 18:57:46,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807526711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807526711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807526711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807526711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:46,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:46,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807526712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,009 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/347c8c049679450d9f318d31b16a4e9c 2023-04-06 18:57:47,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807527016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807527016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807527016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807527016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807527017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,019 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/f29f15d0a2854c5da6576326c9965a48 2023-04-06 18:57:47,345 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/e6ce018744a643b39b90ccec9e7bd0da as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e6ce018744a643b39b90ccec9e7bd0da 2023-04-06 18:57:47,348 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into e6ce018744a643b39b90ccec9e7bd0da(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 18:57:47,348 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:47,348 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807466086; duration=1sec 2023-04-06 18:57:47,348 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:47,427 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/25c6b56f573d490e830aa1163f2a3894 2023-04-06 18:57:47,430 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/347c8c049679450d9f318d31b16a4e9c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c 2023-04-06 18:57:47,433 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c, entries=150, sequenceid=70, filesize=25.1 K 2023-04-06 18:57:47,434 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/f29f15d0a2854c5da6576326c9965a48 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48 2023-04-06 18:57:47,437 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48, entries=150, sequenceid=70, filesize=11.6 K 2023-04-06 18:57:47,438 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/25c6b56f573d490e830aa1163f2a3894 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894 2023-04-06 18:57:47,442 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894, entries=150, sequenceid=70, filesize=11.6 K 2023-04-06 18:57:47,442 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1253ms, sequenceid=70, compaction requested=false 2023-04-06 18:57:47,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:47,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:47,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:47,459 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,459 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,459 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:47,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a2f34a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:47,462 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:47,462 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:47,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:47,463 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,464 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:47,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:47,466 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:47,466 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:47,466 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:47,466 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:47,466 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:47,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:47,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:47,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:47,486 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:47,486 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:47,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:47,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:47,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:47,568 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b549a7cb73a448dea1c3bdc635e70562 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b549a7cb73a448dea1c3bdc635e70562 2023-04-06 18:57:47,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807527573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807527573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807527574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807527575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807527576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,586 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:47,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:47,589 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:47,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:47,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:47,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,590 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,590 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,590 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,590 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:47,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:47,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:47,594 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,594 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:47,594 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:47,594 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:47,594 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,595 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:47,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1236466d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:47,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:47,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,597 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,597 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:47,597 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:47,597 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:47,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:47,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,598 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:47,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:47,600 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:47,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:47,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:47,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:47,600 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:47,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:47,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:47,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,606 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:47,606 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:47,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,606 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807527680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807527680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807527680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807527681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807527681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,706 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:47,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:47,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:47,708 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:47,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:47,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:47,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,710 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:47,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:47,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:47,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:47,711 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,711 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:47,711 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:47,711 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:47,712 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,712 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:47,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:47,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51da1fcd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:47,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:47,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:47,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:47,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:47,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:47,716 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:47,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:47,718 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:47,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:47,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:47,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:47,719 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:47,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,720 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:47,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,723 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,724 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:47,724 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:47,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,824 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:47,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:47,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:47,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:47,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:47,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:47,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:47,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:47,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:47,833 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:47,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:47,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:47,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:47,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:47,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:47,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:47,836 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,836 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:47,836 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:47,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:47,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:47,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:47,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:47,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:47,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2620f0f9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:47,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:47,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:47,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:47,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:47,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:47,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:47,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:47,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:47,847 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:47,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:47,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:47,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:47,847 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:47,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,855 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:47,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:47,859 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:47,859 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:47,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807527883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807527884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807527884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807527884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807527885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,960 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:47,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:47,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:47,965 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:47,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:47,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:47,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,966 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:47,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:47,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:47,967 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,967 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:47,967 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:47,967 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:47,968 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,969 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:47,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209e7fd4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:47,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:47,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,971 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:47,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:47,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:47,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:47,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:47,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:47,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:47,974 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:47,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:47,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:47,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:47,974 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:47,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:47,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:47,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,976 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,977 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:47,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:47,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:47,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:47,979 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:47,979 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:47,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:47,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:47,984 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/303e457ceba84aacacffac1d74338342 2023-04-06 18:57:48,079 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,082 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,086 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb5084[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,091 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,091 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,094 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,097 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,097 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807528189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807528189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807528190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807528190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807528190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,197 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,200 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,201 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,201 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,201 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,201 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,202 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,202 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,202 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,202 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,202 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,203 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,203 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,203 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,203 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e0329[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,204 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,204 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,205 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,206 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,208 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,208 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,208 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,208 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,208 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,209 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,210 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,212 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,212 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,312 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,315 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,315 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,316 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,317 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,317 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,317 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,317 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,317 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,318 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20989cdb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,319 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,320 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,321 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,321 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,323 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,323 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,324 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,327 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,327 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:48,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/04b859bb336e4881b3fb87acc9d61443 2023-04-06 18:57:48,427 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,430 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,431 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,433 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,433 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732b37c3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,434 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,435 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,435 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,435 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,435 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,436 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,436 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,438 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,438 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,439 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,442 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,442 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,443 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,443 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,443 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,543 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,545 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,546 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,547 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,547 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,548 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,549 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,551 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ce7652[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,551 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,552 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,552 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,553 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,553 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,555 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,565 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,565 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,665 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,668 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,668 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,668 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,668 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,669 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,670 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,671 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,671 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,671 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,671 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,672 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b8af09[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,673 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,674 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,675 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,676 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,676 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,676 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,677 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:48,680 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,680 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807528695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807528695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807528696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807528697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807528697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,781 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,783 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,784 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,784 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,785 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,785 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,785 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,786 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,786 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,786 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:48,786 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:48,786 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,787 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,787 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:48,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca29e81[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:48,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:48,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:48,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:48,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:48,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:48,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:48,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,790 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:48,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:48,792 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:48,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:48,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:48,793 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:48,793 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:48,793 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,794 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:48,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:48,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:48,796 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:48,796 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,797 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:48,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:48,797 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:48,797 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:48,797 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:48,797 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/03d8f298de184be8832e71968b3955ae 2023-04-06 18:57:48,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/303e457ceba84aacacffac1d74338342 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342 2023-04-06 18:57:48,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342, entries=200, sequenceid=89, filesize=31.9 K 2023-04-06 18:57:48,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/04b859bb336e4881b3fb87acc9d61443 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443 2023-04-06 18:57:48,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443, entries=150, sequenceid=89, filesize=11.6 K 2023-04-06 18:57:48,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/03d8f298de184be8832e71968b3955ae as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae 2023-04-06 18:57:48,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae, entries=150, sequenceid=89, filesize=11.6 K 2023-04-06 18:57:48,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1276ms, sequenceid=89, compaction requested=true 2023-04-06 18:57:48,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:48,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:48,819 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:48,819 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:48,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:48,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:48,820 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,820 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,820 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e6ce018744a643b39b90ccec9e7bd0da, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=82.3 K 2023-04-06 18:57:48,820 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f32de11d8ee24e4497bbc8483a98a33f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=34.9 K 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e6ce018744a643b39b90ccec9e7bd0da, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464104 2023-04-06 18:57:48,820 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f32de11d8ee24e4497bbc8483a98a33f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464104 2023-04-06 18:57:48,821 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 347c8c049679450d9f318d31b16a4e9c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807465256 2023-04-06 18:57:48,821 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f29f15d0a2854c5da6576326c9965a48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807465256 2023-04-06 18:57:48,821 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 303e457ceba84aacacffac1d74338342, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:48,821 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 04b859bb336e4881b3fb87acc9d61443, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:48,825 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:48,827 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#311 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:48,897 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:48,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:48,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:48,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:48,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:48,900 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:48,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:48,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:48,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,901 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:48,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:48,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:48,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:48,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:48,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:48,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:48,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:48,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:48,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:48,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:48,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:48,902 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:48,902 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:48,912 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a63e1de0989a41c9bf26787dcb7099e0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a63e1de0989a41c9bf26787dcb7099e0 2023-04-06 18:57:48,916 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/c75f4590ef8048d093ceeca75bbbf81d 2023-04-06 18:57:49,235 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/7951c28e93844b5db8cfe3146bd02ff7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/7951c28e93844b5db8cfe3146bd02ff7 2023-04-06 18:57:49,238 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 7951c28e93844b5db8cfe3146bd02ff7(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:49,238 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:49,238 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807468819; duration=0sec 2023-04-06 18:57:49,238 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:49,238 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:49,239 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:49,239 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:49,239 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:49,239 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a426fc4f74444f19afd34a220090da2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=34.9 K 2023-04-06 18:57:49,239 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8a426fc4f74444f19afd34a220090da2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680807464104 2023-04-06 18:57:49,239 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 25c6b56f573d490e830aa1163f2a3894, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680807465256 2023-04-06 18:57:49,240 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 03d8f298de184be8832e71968b3955ae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:49,244 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:49,323 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/d88ce34df7cf4b85a4a21386052af760 2023-04-06 18:57:49,658 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/913fc6f2a6914f9cbfb7764e57baed2e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/913fc6f2a6914f9cbfb7764e57baed2e 2023-04-06 18:57:49,662 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 913fc6f2a6914f9cbfb7764e57baed2e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:49,662 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:49,662 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807468819; duration=0sec 2023-04-06 18:57:49,662 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:49,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807529712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807529713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807529714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807529715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807529717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,732 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/8a6e6a63216641eb96a3681b1bb12427 2023-04-06 18:57:49,736 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/c75f4590ef8048d093ceeca75bbbf81d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d 2023-04-06 18:57:49,739 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d, entries=150, sequenceid=105, filesize=25.1 K 2023-04-06 18:57:49,739 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/d88ce34df7cf4b85a4a21386052af760 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760 2023-04-06 18:57:49,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760, entries=150, sequenceid=105, filesize=11.6 K 2023-04-06 18:57:49,743 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/8a6e6a63216641eb96a3681b1bb12427 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427 2023-04-06 18:57:49,745 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427, entries=150, sequenceid=105, filesize=11.6 K 2023-04-06 18:57:49,746 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 844ms, sequenceid=105, compaction requested=false 2023-04-06 18:57:49,746 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:49,746 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:49,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:49,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161d23f9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:49,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:49,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:49,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:49,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:49,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:49,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:49,751 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:49,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:49,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:49,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:49,751 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:49,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:49,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,752 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,753 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:49,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:49,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:49,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,756 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:49,756 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:49,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:49,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 18:57:49,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:49,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:49,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:49,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:49,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:49,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:49,835 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068c5df0c5adec42759c0c69a6d336564a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068c5df0c5adec42759c0c69a6d336564a 2023-04-06 18:57:49,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807529832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807529833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807529834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807529838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807529839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,856 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:49,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:49,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:49,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:49,859 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:49,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:49,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:49,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:49,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:49,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:49,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:49,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:49,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:49,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:49,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:49,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:49,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:49,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:49,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:49,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,863 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:49,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cbbbb8e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:49,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,866 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:49,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:49,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:49,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:49,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:49,869 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:49,869 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:49,869 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:49,869 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:49,869 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:49,880 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,880 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:49,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,880 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,880 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:49,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:49,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:49,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:49,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:49,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:49,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:49,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,889 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:49,889 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:49,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:49,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807529940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807529940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807529940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807529940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:49,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807529946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,989 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:49,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:49,994 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:49,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:49,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:49,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:49,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:49,998 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51e685f7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,007 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,011 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,011 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,011 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,012 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,012 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,012 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,014 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,016 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,017 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,017 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,047 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/ccb64d5770e94f728153953dc065f0e8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ccb64d5770e94f728153953dc065f0e8 2023-04-06 18:57:50,051 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into ccb64d5770e94f728153953dc065f0e8(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 18:57:50,051 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:50,051 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807468819; duration=1sec 2023-04-06 18:57:50,051 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:50,117 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,120 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,122 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,138 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,142 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,142 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,142 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,142 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807530146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807530146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807530146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807530147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79984fdc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807530152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,160 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,160 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,167 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,167 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,167 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,244 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=126, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/e4774d2f70924f50a1519a980a9ab2dc 2023-04-06 18:57:50,267 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,270 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@665a9fd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,276 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,278 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,278 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,284 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,284 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,283 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,384 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,402 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,403 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,403 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,404 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,409 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,409 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,409 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,414 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,414 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,416 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17131574[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,420 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,420 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,420 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,421 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,421 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,421 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,421 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,421 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,428 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,428 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,428 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,430 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,438 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,439 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,439 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,439 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807530452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807530453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807530453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807530453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807530458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,539 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,541 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,542 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,544 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,544 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,544 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,544 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,546 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72bff35c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,548 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,553 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,553 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,553 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,557 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,558 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/b6bc660855d34b109a84a1f7d133b1c8 2023-04-06 18:57:50,658 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,669 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,669 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,671 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,672 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,672 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,674 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,674 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,688 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,688 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,689 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,689 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e2a478[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,696 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,696 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,706 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,709 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,709 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,714 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,722 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,722 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,722 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,822 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,832 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6263c6f8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,836 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,838 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,838 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,843 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,843 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,943 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:50,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:50,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:50,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:50,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:50,946 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:50,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:50,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:50,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,947 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:50,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:50,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:50,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:50,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:50,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:50,950 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:50,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:50,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd4d775[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:50,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:50,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,954 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:50,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:50,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:50,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:50,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:50,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:50,958 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:50,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:50,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:50,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:50,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:50,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:50,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:50,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:50,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:50,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:50,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:50,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807530958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807530960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:50,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:50,964 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:50,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807530962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:50,966 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:50,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807530962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:50,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807530968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,066 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:51,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:51,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:51,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e554d36637f14da5856d7c689a5260b3 2023-04-06 18:57:51,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:51,082 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:51,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:51,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:51,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:51,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:51,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:51,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:51,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:51,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:51,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:51,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:51,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:51,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:51,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:51,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/e4774d2f70924f50a1519a980a9ab2dc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc 2023-04-06 18:57:51,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:51,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:51,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:51,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fcfc80[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:51,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:51,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc, entries=200, sequenceid=126, filesize=31.9 K 2023-04-06 18:57:51,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/b6bc660855d34b109a84a1f7d133b1c8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8 2023-04-06 18:57:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:51,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:51,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:51,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:51,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8, entries=150, sequenceid=126, filesize=11.6 K 2023-04-06 18:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e554d36637f14da5856d7c689a5260b3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3 2023-04-06 18:57:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:51,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:51,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:51,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:51,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:51,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:51,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:51,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:51,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:51,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:51,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:51,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3, entries=150, sequenceid=126, filesize=11.6 K 2023-04-06 18:57:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1278ms, sequenceid=126, compaction requested=true 2023-04-06 18:57:51,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:51,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:51,101 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:51,101 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:51,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:51,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:51,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:51,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:51,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:51,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:51,101 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:51,102 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:51,102 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84384 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:51,102 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:51,102 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:51,102 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ccb64d5770e94f728153953dc065f0e8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=82.4 K 2023-04-06 18:57:51,102 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:51,102 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:51,102 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,102 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/7951c28e93844b5db8cfe3146bd02ff7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=35.0 K 2023-04-06 18:57:51,103 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ccb64d5770e94f728153953dc065f0e8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:51,103 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7951c28e93844b5db8cfe3146bd02ff7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:51,103 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c75f4590ef8048d093ceeca75bbbf81d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807467572 2023-04-06 18:57:51,103 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d88ce34df7cf4b85a4a21386052af760, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807467572 2023-04-06 18:57:51,103 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e4774d2f70924f50a1519a980a9ab2dc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:51,104 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b6bc660855d34b109a84a1f7d133b1c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:51,108 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:51,142 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#320 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:51,202 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:51,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:51,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:51,216 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:51,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:51,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:51,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:51,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:51,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:51,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:51,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:51,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:51,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:51,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:51,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:51,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:51,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:51,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:51,224 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:51,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:51,550 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/37020b9fcc634b1ba5750aea4fbc8898 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/37020b9fcc634b1ba5750aea4fbc8898 2023-04-06 18:57:51,554 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 37020b9fcc634b1ba5750aea4fbc8898(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:51,554 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:51,554 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807471101; duration=0sec 2023-04-06 18:57:51,554 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:51,554 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:51,556 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:51,556 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:51,556 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:51,556 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/913fc6f2a6914f9cbfb7764e57baed2e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=35.0 K 2023-04-06 18:57:51,559 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 913fc6f2a6914f9cbfb7764e57baed2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680807466401 2023-04-06 18:57:51,560 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8a6e6a63216641eb96a3681b1bb12427, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807467572 2023-04-06 18:57:51,560 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e554d36637f14da5856d7c689a5260b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:51,568 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:51,570 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/dddc54241f9e4e61ba46e4a1e742cb79 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dddc54241f9e4e61ba46e4a1e742cb79 2023-04-06 18:57:51,577 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into dddc54241f9e4e61ba46e4a1e742cb79(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:51,578 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:51,578 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807471101; duration=0sec 2023-04-06 18:57:51,578 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:51,585 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/b8edec4f33084389945efa798e6aafe5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/b8edec4f33084389945efa798e6aafe5 2023-04-06 18:57:51,592 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into b8edec4f33084389945efa798e6aafe5(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:51,593 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:51,593 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807471101; duration=0sec 2023-04-06 18:57:51,593 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:51,650 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ca877adf58cd44068fe41cdec77073b7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ca877adf58cd44068fe41cdec77073b7 2023-04-06 18:57:51,684 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/290802d3dc5b4c50870e6494a51668c5 2023-04-06 18:57:51,739 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/1fd6e6fae1df49368c4375d7cdfdc5bd 2023-04-06 18:57:51,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:51,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:51,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:51,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807531987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:51,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807531994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:51,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807531994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807531995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807531995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807532096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807532099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807532099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807532101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807532101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,197 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/c34f53315fc24a119485bb582326a3f9 2023-04-06 18:57:52,201 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/290802d3dc5b4c50870e6494a51668c5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5 2023-04-06 18:57:52,204 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5, entries=150, sequenceid=140, filesize=25.3 K 2023-04-06 18:57:52,205 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/1fd6e6fae1df49368c4375d7cdfdc5bd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd 2023-04-06 18:57:52,208 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd, entries=150, sequenceid=140, filesize=11.8 K 2023-04-06 18:57:52,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/c34f53315fc24a119485bb582326a3f9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9 2023-04-06 18:57:52,212 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9, entries=150, sequenceid=140, filesize=11.8 K 2023-04-06 18:57:52,213 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 988ms, sequenceid=140, compaction requested=false 2023-04-06 18:57:52,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:52,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c614583[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:52,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,233 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,233 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,241 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,242 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,242 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:52,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:52,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807471993, max timestamp=1680807472307Num uniques 859; 2023-04-06 18:57:52,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:52,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807471993, max timestamp=1680807472307Num uniques 859; 2023-04-06 18:57:52,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:52,310 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:52,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807471993, max timestamp=1680807472307Num uniques 859; 2023-04-06 18:57:52,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:52,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,342 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:52,342 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d6d3a1010f9c46b6a9c5b865a974ecf5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d6d3a1010f9c46b6a9c5b865a974ecf5 2023-04-06 18:57:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:52,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:52,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:52,350 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:52,350 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,350 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:52,350 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807472329, max timestamp=1680807472349Num uniques 859; 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807472329, max timestamp=1680807472349Num uniques 859; 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807472329, max timestamp=1680807472349Num uniques 859; 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:52,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:57:52,351 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,351 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,352 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:52,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,354 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/80c0795605bd4a4fb490e1f595678c08 2023-04-06 18:57:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:52,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:52,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:52,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:52,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:52,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:52,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:52,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/39206994325b42d2b47d2dd49ca893e0 2023-04-06 18:57:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c900f1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,382 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,383 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,383 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,389 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,389 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807472373, max timestamp=1680807472387Num uniques 1068; 2023-04-06 18:57:52,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:57:52,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:52,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807472373, max timestamp=1680807472388Num uniques 1068; 2023-04-06 18:57:52,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807472373, max timestamp=1680807472387Num uniques 1068; 2023-04-06 18:57:52,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:57:52,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:57:52,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807532422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807472413, max timestamp=1680807472428Num uniques 1077; 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807472413, max timestamp=1680807472428Num uniques 1077; 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807472413, max timestamp=1680807472428Num uniques 1077; 2023-04-06 18:57:52,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 18:57:52,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807532468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807472455, max timestamp=1680807472479Num uniques 1086; 2023-04-06 18:57:52,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:52,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807472455, max timestamp=1680807472479Num uniques 1086; 2023-04-06 18:57:52,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:52,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807472455, max timestamp=1680807472479Num uniques 1086; 2023-04-06 18:57:52,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:57:52,489 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:52,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:52,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:52,492 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:52,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:52,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:52,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,493 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,493 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:52,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:52,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:52,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:52,495 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,495 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:52,495 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:52,495 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed18e2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,498 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,498 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,500 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:52,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,504 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,504 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,506 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,508 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,508 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:52,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807472510, max timestamp=1680807472524Num uniques 1045; 2023-04-06 18:57:52,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:57:52,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807472510, max timestamp=1680807472524Num uniques 1045; 2023-04-06 18:57:52,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:57:52,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807472510, max timestamp=1680807472524Num uniques 1045; 2023-04-06 18:57:52,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:57:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807532556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807532557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807532560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807532565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807532582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,609 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:52,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:52,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:52,614 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:52,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:52,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:52,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:52,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:52,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:52,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:52,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:52,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:52,629 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:52,629 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:52,630 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,630 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:52,630 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:52,630 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,632 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,632 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,633 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cf2fde3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:52,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,642 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,642 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,644 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,645 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,656 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,656 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807532666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807532666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807532666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807532676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,756 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:52,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:52,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:52,758 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:52,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:52,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:52,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,759 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:52,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:52,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:52,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:52,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,761 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:52,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:52,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:52,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:52,766 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,772 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a09eca3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,774 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,774 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,779 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,790 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,790 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,790 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807532788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/293279b8e29b40758f9257e2465f6adb 2023-04-06 18:57:52,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/80c0795605bd4a4fb490e1f595678c08 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08 2023-04-06 18:57:52,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08, entries=250, sequenceid=166, filesize=38.9 K 2023-04-06 18:57:52,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/39206994325b42d2b47d2dd49ca893e0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0 2023-04-06 18:57:52,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0, entries=150, sequenceid=166, filesize=11.8 K 2023-04-06 18:57:52,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/293279b8e29b40758f9257e2465f6adb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb 2023-04-06 18:57:52,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb, entries=150, sequenceid=166, filesize=11.8 K 2023-04-06 18:57:52,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 505ms, sequenceid=166, compaction requested=true 2023-04-06 18:57:52,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:52,815 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:52,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:52,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:52,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:57:52,816 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:52,817 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,817 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dddc54241f9e4e61ba46e4a1e742cb79, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=89.7 K 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting dddc54241f9e4e61ba46e4a1e742cb79, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:52,817 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,817 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/37020b9fcc634b1ba5750aea4fbc8898, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=35.4 K 2023-04-06 18:57:52,817 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 290802d3dc5b4c50870e6494a51668c5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680807469833 2023-04-06 18:57:52,818 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 37020b9fcc634b1ba5750aea4fbc8898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:52,818 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 80c0795605bd4a4fb490e1f595678c08, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:52,818 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1fd6e6fae1df49368c4375d7cdfdc5bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680807469833 2023-04-06 18:57:52,819 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 39206994325b42d2b47d2dd49ca893e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:52,835 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:52,872 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#329 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:52,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:52,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:52,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:52,885 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/e0279911ed3746988983fdb252b76207 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0279911ed3746988983fdb252b76207 2023-04-06 18:57:52,890 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into e0279911ed3746988983fdb252b76207(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:52,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:52,890 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807472816; duration=0sec 2023-04-06 18:57:52,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:52,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807532881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807532884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807532884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:52,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:52,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/b8edec4f33084389945efa798e6aafe5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=35.4 K 2023-04-06 18:57:52,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b8edec4f33084389945efa798e6aafe5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1680807469714 2023-04-06 18:57:52,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c34f53315fc24a119485bb582326a3f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680807469833 2023-04-06 18:57:52,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 293279b8e29b40758f9257e2465f6adb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:52,894 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:52,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:52,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:52,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:52,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:52,897 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:52,897 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,897 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:52,897 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:52,897 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,897 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,899 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,899 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,900 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:52,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:52,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:52,902 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:52,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:52,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:52,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807532891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:52,903 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:52,903 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:52,904 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,906 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:52,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32cfbd9b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:52,906 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:52,906 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,906 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,907 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:52,907 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,907 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:52,907 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:52,907 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:52,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:52,908 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,908 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:52,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:52,910 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:52,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:52,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:52,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:52,910 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:52,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:52,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,917 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:52,917 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:52,918 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:52,918 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:52,918 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:52,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:52,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:52,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:52,918 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:52,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:52,919 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:52,919 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:52,919 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:52,926 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:52,968 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/715a9c9553ea420c9261127eec13834b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/715a9c9553ea420c9261127eec13834b 2023-04-06 18:57:52,972 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 715a9c9553ea420c9261127eec13834b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:52,972 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:52,972 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807472816; duration=0sec 2023-04-06 18:57:52,972 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:52,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807532995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807532999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807533005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,019 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,026 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,027 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,028 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,028 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,028 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,030 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51868928[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,032 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,032 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,032 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,033 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,033 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,033 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,036 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,036 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,037 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,039 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,041 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,040 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,040 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,041 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,041 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,041 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,041 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,041 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807533098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,141 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,144 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,144 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,145 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,145 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,153 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,153 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,153 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,153 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,163 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,164 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,164 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a006b1e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,167 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,167 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,168 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,170 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,170 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,171 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,172 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,173 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,174 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,174 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807533193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807533199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807533208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807533211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,274 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,281 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,298 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,312 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,312 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,312 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,312 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,312 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,314 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,315 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,315 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f99fa09[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,317 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040675c31659f7f649bc910e256502a71f6d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040675c31659f7f649bc910e256502a71f6d 2023-04-06 18:57:53,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,318 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,319 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,320 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,320 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,320 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,320 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,321 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,321 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,321 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,325 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,325 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,375 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/9f9e20a614e3430dad61c01569b677b8 2023-04-06 18:57:53,426 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,430 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,430 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,431 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,431 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,432 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,440 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,442 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7075ab04[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,456 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,458 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,458 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,462 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,464 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,464 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,464 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807533508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807533519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807533530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,564 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,570 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,573 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,573 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,573 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,573 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,574 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,575 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,575 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155fa3eb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,578 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,578 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,578 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,580 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,580 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,580 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,581 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,581 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,581 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,583 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,585 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,585 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42968 deadline: 1680807533608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,685 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,691 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,691 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,692 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,693 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,693 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,693 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,693 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4e72c5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,697 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,697 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,697 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,699 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,699 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:53,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43056 deadline: 1680807533698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,700 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,703 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,703 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,703 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,781 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/a208d8a8a4344d1da7a6fa3d7d003c67 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a208d8a8a4344d1da7a6fa3d7d003c67 2023-04-06 18:57:53,785 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into a208d8a8a4344d1da7a6fa3d7d003c67(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:53,785 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:53,785 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807472815; duration=0sec 2023-04-06 18:57:53,785 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:53,804 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,807 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,808 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,808 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,811 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7792f7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,815 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,816 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,816 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,821 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,822 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,822 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,822 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6d9823f3d86f4894a9393cccc7eae034 2023-04-06 18:57:53,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,822 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,822 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:53,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:53,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,922 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:53,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:53,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:53,925 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:53,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:53,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:53,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,925 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,926 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:53,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:53,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:53,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:53,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:53,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:53,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:53,931 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:53,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:53,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:53,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:53,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:53,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:53,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@718721f6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:53,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:53,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,934 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:53,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:53,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:53,934 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:53,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,936 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:53,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:53,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:53,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:53,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:53,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:53,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:53,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:53,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:53,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:53,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:53,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:53,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:53,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:53,945 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:53,945 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:54,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807534016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807534029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:54,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807534037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,045 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:54,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:54,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:54,050 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:54,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:54,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:54,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:54,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:54,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:54,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:54,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:54,054 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:54,055 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,055 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:54,055 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:54,055 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:54,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,056 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,056 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:54,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19af94d6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:54,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:54,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,058 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:54,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:54,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:54,058 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:54,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:54,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:54,062 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:54,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:54,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:54,062 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:54,062 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:54,064 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,064 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,064 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,065 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,067 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:54,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:54,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:54,069 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:54,069 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:54,072 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,072 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,072 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,169 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:54,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:54,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:54,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:54,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:54,178 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:54,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:54,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:54,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,182 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:54,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:54,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:54,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:54,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:54,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:54,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:54,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:54,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:54,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:54,197 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:54,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d8157c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:54,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:54,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:54,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:54,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:54,208 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:54,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:54,213 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:54,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:54,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:54,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:54,213 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:54,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,231 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:54,231 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:54,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/c55e1b1fe08946468c92e8e2da9f78ff 2023-04-06 18:57:54,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/9f9e20a614e3430dad61c01569b677b8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8 2023-04-06 18:57:54,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8, entries=200, sequenceid=272, filesize=32.3 K 2023-04-06 18:57:54,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6d9823f3d86f4894a9393cccc7eae034 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034 2023-04-06 18:57:54,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034, entries=150, sequenceid=272, filesize=11.9 K 2023-04-06 18:57:54,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/c55e1b1fe08946468c92e8e2da9f78ff as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff 2023-04-06 18:57:54,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff, entries=150, sequenceid=272, filesize=11.9 K 2023-04-06 18:57:54,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1380ms, sequenceid=272, compaction requested=false 2023-04-06 18:57:54,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:54,332 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:54,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:54,335 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:54,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:54,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:54,336 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,336 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,336 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,336 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:54,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:54,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:54,343 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:54,343 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:54,344 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:54,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:54,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:54,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807474647, max timestamp=1680807474720Num uniques 859; 2023-04-06 18:57:54,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:54,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807474647, max timestamp=1680807474720Num uniques 859; 2023-04-06 18:57:54,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:54,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807474647, max timestamp=1680807474720Num uniques 859; 2023-04-06 18:57:54,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:57:54,757 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040674895a9d832d4927aeacdb5067c8c7b3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040674895a9d832d4927aeacdb5067c8c7b3 2023-04-06 18:57:54,764 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/dab47d52b3e7410d9608bae21681bbfb 2023-04-06 18:57:54,791 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6157941dc7e8412eb0399108bd9f7024 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807474745, max timestamp=1680807474804Num uniques 1018; 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807474745, max timestamp=1680807474804Num uniques 1018; 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807474745, max timestamp=1680807474804Num uniques 1018; 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-06 18:57:54,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-06 18:57:54,816 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/1653f7a7e39d4947aad49598c97244fe 2023-04-06 18:57:54,819 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/dab47d52b3e7410d9608bae21681bbfb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb 2023-04-06 18:57:54,824 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb, entries=150, sequenceid=282, filesize=25.4 K 2023-04-06 18:57:54,825 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6157941dc7e8412eb0399108bd9f7024 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024 2023-04-06 18:57:54,836 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024, entries=150, sequenceid=282, filesize=11.9 K 2023-04-06 18:57:54,837 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/1653f7a7e39d4947aad49598c97244fe as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe 2023-04-06 18:57:54,841 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe, entries=150, sequenceid=282, filesize=11.9 K 2023-04-06 18:57:54,841 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=87.22 KB/89310 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 497ms, sequenceid=282, compaction requested=true 2023-04-06 18:57:54,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:54,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:54,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:54,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:54,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:54,843 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:54,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2628c97a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:54,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:54,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:54,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:54,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=93.93 KB heapSize=259.92 KB 2023-04-06 18:57:54,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:54,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:54,846 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:54,847 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:57:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:54,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:54,850 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:54,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:54,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:54,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:54,850 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:54,863 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,863 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,863 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,864 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:54,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,872 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:54,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,872 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:54,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807534891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807534894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,972 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:54,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:54,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:54,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:54,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:54,974 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:54,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:54,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:54,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,975 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,975 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,975 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:54,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:54,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:54,976 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,976 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:54,976 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:54,976 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:54,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:54,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:54,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:54,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee8c4d8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:54,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:54,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:54,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,981 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:54,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:54,983 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:54,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:54,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:54,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:54,983 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:54,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:54,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:54,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:54,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:54,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:54,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:54,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:54,989 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:54,989 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:54,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:54,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:54,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:54,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807534998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807534998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43040 deadline: 1680807535029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807535044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42990 deadline: 1680807535049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,089 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,091 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,094 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,094 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,094 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,094 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b84f4f7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,098 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,100 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,100 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,104 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,104 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,204 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,206 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,207 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,210 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d81630e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807535206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807535207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,215 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,215 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,219 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,219 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,262 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d52bf765bbc4493aa1bbf00675d8fbc4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d52bf765bbc4493aa1bbf00675d8fbc4 2023-04-06 18:57:55,320 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,323 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,324 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,325 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,333 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,333 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,333 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,333 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,333 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0206dd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,340 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,341 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,341 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,341 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,341 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,341 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,343 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,343 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,344 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,345 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,347 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,347 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,347 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,347 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,348 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,348 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,449 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,451 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,452 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,454 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,454 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,454 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,454 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,454 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3564cfa1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,460 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,460 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,461 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,461 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,461 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,461 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,462 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,465 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,465 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,465 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807535514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807535514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,565 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,570 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,578 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,578 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,578 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,578 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,579 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ebb6b6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,588 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,588 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,588 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,589 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,591 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,591 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,593 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,595 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,596 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,596 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,669 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/ae323afaa865442bb62e6f61d6ccd5fa 2023-04-06 18:57:55,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/b524fa9af3bc44a3b3a1d6dde9b6c838 2023-04-06 18:57:55,696 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,710 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,712 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f843f96[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,737 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,740 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,740 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,747 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,748 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,750 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,750 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,750 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,751 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,751 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,751 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,851 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,854 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,854 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:55,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:55,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:55,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:55,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:55,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:55,860 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,860 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:55,860 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:55,860 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:55,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:55,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:55,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62bdf855[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:55,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,874 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:55,874 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:55,874 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:55,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:55,875 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:55,875 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,875 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:55,875 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:55,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:55,887 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:55,887 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:55,887 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:55,887 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:55,887 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:55,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:55,890 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:55,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:55,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:55,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:55,891 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:55,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,892 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:55,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:55,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:55,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:55,992 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:55,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:55,995 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:55,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:55,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:55,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:55,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:55,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:55,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:55,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:55,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:56,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:56,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:56,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:56,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:56,002 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:56,003 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:56,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:56,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:56,003 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:56,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:56,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71373a64[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:56,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:56,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,007 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,007 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:56,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:56,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:56,007 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:56,008 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:56,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:56,012 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:56,012 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:56,012 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:56,012 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:56,012 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:56,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,014 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,015 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,016 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:56,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:56,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:56,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,016 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:56,016 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:56,016 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:56,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:56,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807536021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:56,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807536022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/f5f0e242962c487eb717f6ed00810b70 2023-04-06 18:57:56,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/ae323afaa865442bb62e6f61d6ccd5fa as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa 2023-04-06 18:57:56,116 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:56,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa, entries=250, sequenceid=330, filesize=39.2 K 2023-04-06 18:57:56,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/b524fa9af3bc44a3b3a1d6dde9b6c838 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838 2023-04-06 18:57:56,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:56,119 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:56,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:56,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:56,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:56,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,120 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:56,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:56,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:56,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:56,121 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:56,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:56,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:56,122 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:56,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:56,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,123 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352d6cf2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:56,124 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:56,124 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,125 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:56,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:56,125 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:56,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,125 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:56,125 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/f5f0e242962c487eb717f6ed00810b70 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70 2023-04-06 18:57:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:56,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:56,128 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:56,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:56,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:56,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:56,128 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:56,129 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,129 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:56,129 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:56,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:56,131 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70, entries=150, sequenceid=330, filesize=11.9 K 2023-04-06 18:57:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:56,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~279.80 KB/286512, currentSize=80.51 KB/82440 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1292ms, sequenceid=330, compaction requested=true 2023-04-06 18:57:56,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:56,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:56,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:56,139 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,139 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:56,139 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:56,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:56,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,140 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:56,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:56,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:56,140 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:56,140 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,140 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:56,141 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0279911ed3746988983fdb252b76207, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=47.9 K 2023-04-06 18:57:56,141 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:56,141 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125568 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:56,141 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:56,141 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,141 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a208d8a8a4344d1da7a6fa3d7d003c67, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=122.6 K 2023-04-06 18:57:56,141 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e0279911ed3746988983fdb252b76207, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:56,141 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a208d8a8a4344d1da7a6fa3d7d003c67, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6d9823f3d86f4894a9393cccc7eae034, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680807472555 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9f9e20a614e3430dad61c01569b677b8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680807472555 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6157941dc7e8412eb0399108bd9f7024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680807472878 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting dab47d52b3e7410d9608bae21681bbfb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680807472878 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b524fa9af3bc44a3b3a1d6dde9b6c838, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474837 2023-04-06 18:57:56,142 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ae323afaa865442bb62e6f61d6ccd5fa, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474822 2023-04-06 18:57:56,151 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:56,153 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#341 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:56,202 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/157c6269ecee4db8a3b9dc71950f30f4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/157c6269ecee4db8a3b9dc71950f30f4 2023-04-06 18:57:56,205 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 157c6269ecee4db8a3b9dc71950f30f4(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:56,205 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:56,205 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=12, startTime=1680807476139; duration=0sec 2023-04-06 18:57:56,206 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:56,206 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:57:56,207 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:57:56,207 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:56,207 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,207 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/715a9c9553ea420c9261127eec13834b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=47.9 K 2023-04-06 18:57:56,207 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 715a9c9553ea420c9261127eec13834b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1680807471993 2023-04-06 18:57:56,210 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c55e1b1fe08946468c92e8e2da9f78ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1680807472555 2023-04-06 18:57:56,210 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 1653f7a7e39d4947aad49598c97244fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1680807472878 2023-04-06 18:57:56,211 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f5f0e242962c487eb717f6ed00810b70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474837 2023-04-06 18:57:56,220 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#343 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:56,241 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:56,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:56,244 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:56,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:56,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:56,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:56,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,246 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:56,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:56,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:56,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:56,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:56,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:56,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:56,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:56,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:56,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:56,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:56,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:56,247 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:56,247 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:56,248 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:56,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:56,565 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/df1f3c05ad8543d1935891ea5bd5f201 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/df1f3c05ad8543d1935891ea5bd5f201 2023-04-06 18:57:56,570 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into df1f3c05ad8543d1935891ea5bd5f201(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:56,570 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:56,570 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=12, startTime=1680807476139; duration=0sec 2023-04-06 18:57:56,570 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:56,637 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/1c9fd57678c14d009949a9dd95440f9d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1c9fd57678c14d009949a9dd95440f9d 2023-04-06 18:57:56,645 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 1c9fd57678c14d009949a9dd95440f9d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:56,645 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:56,645 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=12, startTime=1680807476139; duration=0sec 2023-04-06 18:57:56,645 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:56,674 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b890a077d5214956828d89b1ffeca60d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b890a077d5214956828d89b1ffeca60d 2023-04-06 18:57:56,687 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/a589c3be184b49e9b99b11fc1287e251 2023-04-06 18:57:56,707 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/78683d2ac8354a568bc5cf581c0e5999 2023-04-06 18:57:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:57,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807537053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807537053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43016 deadline: 1680807537054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807537054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807537055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,063 DEBUG [Thread-2269] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:57:57,148 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/de3fcc939d014b569a510c28f13be872 2023-04-06 18:57:57,152 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/a589c3be184b49e9b99b11fc1287e251 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251 2023-04-06 18:57:57,154 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251, entries=150, sequenceid=346, filesize=25.4 K 2023-04-06 18:57:57,155 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/78683d2ac8354a568bc5cf581c0e5999 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999 2023-04-06 18:57:57,158 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999, entries=150, sequenceid=346, filesize=11.9 K 2023-04-06 18:57:57,159 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/de3fcc939d014b569a510c28f13be872 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872 2023-04-06 18:57:57,161 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872, entries=150, sequenceid=346, filesize=11.9 K 2023-04-06 18:57:57,162 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 915ms, sequenceid=346, compaction requested=false 2023-04-06 18:57:57,162 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:57,162 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,163 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9b9d4e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:57,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 18:57:57,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:57,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:57,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:57,166 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:57,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:57,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:57,167 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,174 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,174 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,180 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,181 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,182 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,183 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,183 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,183 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807537177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807537178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807537179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807537180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,283 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,286 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,286 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,287 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,287 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,287 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,287 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,288 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,289 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807537285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e9ad38[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,290 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,291 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,292 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,292 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,292 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,292 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807537286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807537286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807537286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,294 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,294 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,294 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,295 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,296 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,298 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:57,297 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,297 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,298 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,298 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,298 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,298 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,398 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,400 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,400 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,402 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,403 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,403 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,403 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,403 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,403 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,404 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,404 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,405 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec54d9b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,406 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,406 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,406 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,406 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,406 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,406 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,406 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,407 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,407 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,407 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,409 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,409 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,409 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,409 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,409 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,410 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,410 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,410 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,410 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,412 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,422 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,422 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807537490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807537494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807537495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807537495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,522 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,524 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,524 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,525 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,525 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,530 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,539 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,539 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,539 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,539 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,539 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,540 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,540 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,541 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,541 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d1fc4d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,542 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,542 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,543 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,543 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,544 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,552 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,552 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,552 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,566 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,566 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,566 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,571 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,571 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,578 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406736acee33d16465c9ecfef841e8c4eda to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406736acee33d16465c9ecfef841e8c4eda 2023-04-06 18:57:57,605 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/7cf2ae44ae704831b61aafc85ca0e74b 2023-04-06 18:57:57,672 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,678 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,679 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,681 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,681 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,681 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,681 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,682 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47876cf4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,694 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,694 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,697 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,697 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,701 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,702 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,702 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807537795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,802 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,804 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807537803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807537804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807537805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,814 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,814 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,814 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,814 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,822 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,822 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,823 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,823 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbc8951[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,825 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,825 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,825 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,825 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,825 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,826 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,828 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,830 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,832 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,832 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:57,932 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:57,935 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:57,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:57,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:57,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:57,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:57,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:57,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:57,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:57,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:57,937 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:57,937 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:57,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:57,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6801aac8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:57,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:57,939 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,940 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,940 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:57,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:57,940 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:57,940 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:57,941 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:57,941 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,941 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,941 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:57,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:57,943 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:57,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:57,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:57,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:57,943 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:57,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:57,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:57,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:57,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:57,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:57,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:57,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:57,949 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:57,949 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:57,949 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:57,949 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:57,949 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:57,949 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:58,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/019b61d1644a423d87300b444547fba2 2023-04-06 18:57:58,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/d83afd17fbbe4e64bc55c0d97907cb30 2023-04-06 18:57:58,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/7cf2ae44ae704831b61aafc85ca0e74b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b 2023-04-06 18:57:58,049 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:58,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b, entries=150, sequenceid=367, filesize=25.4 K 2023-04-06 18:57:58,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:58,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/019b61d1644a423d87300b444547fba2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2 2023-04-06 18:57:58,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:58,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:58,052 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:58,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:58,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:58,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:58,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:58,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:58,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 18:57:58,053 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:58,053 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:58,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/d83afd17fbbe4e64bc55c0d97907cb30 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30 2023-04-06 18:57:58,054 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:58,054 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:58,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:58,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:58,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:58,060 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:58,060 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:58,060 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:58,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30, entries=150, sequenceid=367, filesize=11.9 K 2023-04-06 18:57:58,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 897ms, sequenceid=367, compaction requested=true 2023-04-06 18:57:58,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:58,062 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:58,062 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:58,062 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:58,063 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78684 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:58,063 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:57:58,063 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:58,063 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:58,063 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/157c6269ecee4db8a3b9dc71950f30f4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=76.8 K 2023-04-06 18:57:58,063 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:57:58,063 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:58,063 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/df1f3c05ad8543d1935891ea5bd5f201, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.2 K 2023-04-06 18:57:58,064 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 157c6269ecee4db8a3b9dc71950f30f4, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474837 2023-04-06 18:57:58,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting df1f3c05ad8543d1935891ea5bd5f201, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474837 2023-04-06 18:57:58,064 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a589c3be184b49e9b99b11fc1287e251, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680807474873 2023-04-06 18:57:58,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:58,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:58,078 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7cf2ae44ae704831b61aafc85ca0e74b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:57:58,078 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 78683d2ac8354a568bc5cf581c0e5999, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680807474873 2023-04-06 18:57:58,079 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 019b61d1644a423d87300b444547fba2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:57:58,084 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:58,084 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:58,084 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:57:58,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:58,085 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,085 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:58,086 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,115 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#351 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:58,132 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069fc5bdd70f6c4ed6bd3fd776849c0a70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069fc5bdd70f6c4ed6bd3fd776849c0a70 2023-04-06 18:57:58,138 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/c1ea5f43c2de4728bdc9c108e747b6a3 2023-04-06 18:57:58,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:58,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:58,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807538369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807538376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807538381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807538382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807538483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807538487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807538494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807538494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,528 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/9b70839baed24b9ab9751eb441b90ee4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/9b70839baed24b9ab9751eb441b90ee4 2023-04-06 18:57:58,533 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 9b70839baed24b9ab9751eb441b90ee4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:58,533 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:58,533 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807478062; duration=0sec 2023-04-06 18:57:58,533 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:57:58,533 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:57:58,534 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:57:58,534 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:57:58,535 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:58,535 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1c9fd57678c14d009949a9dd95440f9d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.2 K 2023-04-06 18:57:58,535 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1c9fd57678c14d009949a9dd95440f9d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680807474837 2023-04-06 18:57:58,535 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting de3fcc939d014b569a510c28f13be872, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680807474873 2023-04-06 18:57:58,536 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d83afd17fbbe4e64bc55c0d97907cb30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:57:58,542 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:57:58,558 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/dbbafc18007f4589b177435e4c16d283 2023-04-06 18:57:58,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807538688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807538694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807538704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807538706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,946 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/e37ea8bacf404f3788b6184f3b9366db as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e37ea8bacf404f3788b6184f3b9366db 2023-04-06 18:57:58,950 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into e37ea8bacf404f3788b6184f3b9366db(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:58,950 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:58,951 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807478062; duration=0sec 2023-04-06 18:57:58,951 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:58,970 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/393c3ad815754ceba01105833f5170d2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/393c3ad815754ceba01105833f5170d2 2023-04-06 18:57:58,971 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e455968589e34c9688be62e47b48dc02 2023-04-06 18:57:58,974 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 393c3ad815754ceba01105833f5170d2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:57:58,974 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:58,974 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807478062; duration=0sec 2023-04-06 18:57:58,974 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:57:58,974 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/c1ea5f43c2de4728bdc9c108e747b6a3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3 2023-04-06 18:57:58,978 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3, entries=100, sequenceid=380, filesize=18.6 K 2023-04-06 18:57:58,979 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/dbbafc18007f4589b177435e4c16d283 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283 2023-04-06 18:57:58,981 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283, entries=100, sequenceid=380, filesize=9.5 K 2023-04-06 18:57:58,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/e455968589e34c9688be62e47b48dc02 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02 2023-04-06 18:57:58,988 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02, entries=100, sequenceid=380, filesize=9.5 K 2023-04-06 18:57:58,988 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 904ms, sequenceid=380, compaction requested=false 2023-04-06 18:57:58,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:57:58,989 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:58,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:58,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:58,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:58,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:58,990 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dd1011[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:58,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:58,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:58,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:58,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:58,994 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:58,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:58,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:58,995 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:58,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 18:57:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:57:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:57:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:57:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:57:58,998 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:58,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:58,999 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:58,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:58,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:58,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:58,999 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,015 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,017 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,024 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,024 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,024 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,024 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:59,024 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,024 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,025 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,024 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,025 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,025 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,025 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,025 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807539019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807539020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807539021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807539028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,124 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,127 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,127 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,128 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,128 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b413cd2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,131 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,131 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,132 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,132 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,132 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,132 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807539129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807539129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,133 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,133 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807539129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,133 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,135 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,135 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807539134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,141 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,141 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,141 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,241 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,243 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,246 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,246 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,246 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,246 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,247 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63afc4ea[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,250 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,250 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,252 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,252 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,252 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,253 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,265 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,265 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,266 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807539334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807539334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807539334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807539341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,366 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,368 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,369 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,371 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,372 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,372 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d75bd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,376 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,376 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,376 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,397 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,397 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,431 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406265419c0eb5c48b8b33fefeb41b8a18e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406265419c0eb5c48b8b33fefeb41b8a18e 2023-04-06 18:57:59,497 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,500 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69afa0f7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,508 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,508 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,512 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,512 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,612 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,615 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,618 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,618 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,618 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7aa6bc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,620 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,621 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,623 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,623 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,626 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,628 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,628 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807539639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807539640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807539641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:57:59,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807539645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,728 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,730 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,733 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,733 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,734 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,734 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,734 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,734 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,734 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,735 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6efb9ad[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,737 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,737 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,737 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,737 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,738 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,740 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,740 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,741 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,741 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,741 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,741 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,744 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:59,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,744 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,744 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,835 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=403, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/7ce0800a7ccf4b6ba17165682e3564db 2023-04-06 18:57:59,844 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,847 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,848 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,848 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,849 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,849 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,850 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,851 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b329450[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,852 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,852 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,853 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,853 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,853 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,855 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,855 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,855 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,855 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,855 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:57:59,858 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,858 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,958 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:57:59,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:57:59,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:57:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:57:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:57:59,961 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:57:59,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:57:59,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:57:59,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:57:59,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:57:59,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:57:59,964 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,964 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:57:59,964 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:57:59,964 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:57:59,964 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:57:59,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,965 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:57:59,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:57:59,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8e29d9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:57:59,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:57:59,967 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,967 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:57:59,967 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:57:59,967 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,968 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:57:59,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:57:59,969 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:57:59,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:57:59,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:57:59,969 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:57:59,969 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:57:59,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:57:59,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,970 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:57:59,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:57:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:57:59,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:57:59,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:57:59,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:57:59,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:57:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:57:59,973 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:57:59,973 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:57:59,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,073 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:00,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:00,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:00,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:00,076 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:00,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:00,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:00,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:00,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:00,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:00,079 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,079 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:00,079 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:00,079 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:00,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:00,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,080 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:00,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:00,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310f2fd2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:00,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,082 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:00,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:00,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:00,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:00,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:00,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:00,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:00,085 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:00,085 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:00,085 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:00,085 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:00,085 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:00,085 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,085 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,085 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:00,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:00,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,089 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,089 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,089 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:00,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807540143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807540146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807540146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807540151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,189 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:00,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:00,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:00,191 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,192 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,193 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:00,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:00,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:00,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:00,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:00,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:00,193 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:00,193 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,194 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:00,194 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:00,194 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:00,194 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,194 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,195 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,195 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:00,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:00,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a56ca8f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:00,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,196 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:00,196 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:00,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:00,196 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:00,197 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:00,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:00,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:00,198 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:00,198 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:00,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:00,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:00,199 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:00,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,202 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:00,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:00,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,202 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,202 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:00,202 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:00,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/869e6e7250454ecb9188063cc4ba9858 2023-04-06 18:58:00,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/9b98dc5fb8d74645a20167cab797c7e8 2023-04-06 18:58:00,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/7ce0800a7ccf4b6ba17165682e3564db as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db 2023-04-06 18:58:00,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db, entries=200, sequenceid=403, filesize=32.3 K 2023-04-06 18:58:00,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/869e6e7250454ecb9188063cc4ba9858 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858 2023-04-06 18:58:00,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858, entries=150, sequenceid=403, filesize=11.9 K 2023-04-06 18:58:00,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/9b98dc5fb8d74645a20167cab797c7e8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8 2023-04-06 18:58:00,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8, entries=150, sequenceid=403, filesize=11.9 K 2023-04-06 18:58:00,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1266ms, sequenceid=403, compaction requested=true 2023-04-06 18:58:00,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:00,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:00,263 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:00,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:00,264 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:00,264 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:00,264 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78778 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:00,264 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:58:00,264 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,264 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:00,264 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e37ea8bacf404f3788b6184f3b9366db, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=76.9 K 2023-04-06 18:58:00,265 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:58:00,265 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,265 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/9b70839baed24b9ab9751eb441b90ee4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=33.9 K 2023-04-06 18:58:00,265 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e37ea8bacf404f3788b6184f3b9366db, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:58:00,265 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9b70839baed24b9ab9751eb441b90ee4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:58:00,265 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c1ea5f43c2de4728bdc9c108e747b6a3, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680807477178 2023-04-06 18:58:00,266 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting dbbafc18007f4589b177435e4c16d283, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680807477178 2023-04-06 18:58:00,266 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7ce0800a7ccf4b6ba17165682e3564db, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:00,266 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 869e6e7250454ecb9188063cc4ba9858, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:00,272 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:00,273 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#359 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:00,302 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:00,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:00,305 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:00,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:00,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:00,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,306 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:00,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:00,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:00,307 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,307 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:00,307 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:00,308 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:00,688 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/fb118d13658547e3a6d726cb4459c72c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/fb118d13658547e3a6d726cb4459c72c 2023-04-06 18:58:00,692 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into fb118d13658547e3a6d726cb4459c72c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:00,692 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:00,692 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807480263; duration=0sec 2023-04-06 18:58:00,692 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:00,692 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:00,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:00,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:58:00,693 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,693 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/393c3ad815754ceba01105833f5170d2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=33.9 K 2023-04-06 18:58:00,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 393c3ad815754ceba01105833f5170d2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1680807477052 2023-04-06 18:58:00,694 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting e455968589e34c9688be62e47b48dc02, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680807477178 2023-04-06 18:58:00,694 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9b98dc5fb8d74645a20167cab797c7e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:00,699 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#362 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:00,707 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/2adf52cb7c44455f9178a6aa74d0d219 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2adf52cb7c44455f9178a6aa74d0d219 2023-04-06 18:58:00,711 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 2adf52cb7c44455f9178a6aa74d0d219(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:00,711 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:00,711 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807480264; duration=0sec 2023-04-06 18:58:00,711 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:00,726 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b2dcccd1666d4d6597b4baeacbef5595 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b2dcccd1666d4d6597b4baeacbef5595 2023-04-06 18:58:00,752 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/460e7b538e834b9d90e1ae56d42fb199 2023-04-06 18:58:00,772 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/cf8656350ec64623a2f5ff4fe57d874b 2023-04-06 18:58:00,785 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/655b6083aaf84f1a873a4c359542e82e 2023-04-06 18:58:00,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/460e7b538e834b9d90e1ae56d42fb199 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199 2023-04-06 18:58:00,793 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199, entries=150, sequenceid=415, filesize=25.4 K 2023-04-06 18:58:00,794 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/cf8656350ec64623a2f5ff4fe57d874b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b 2023-04-06 18:58:00,798 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b, entries=150, sequenceid=415, filesize=11.9 K 2023-04-06 18:58:00,799 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/655b6083aaf84f1a873a4c359542e82e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e 2023-04-06 18:58:00,801 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e, entries=150, sequenceid=415, filesize=11.9 K 2023-04-06 18:58:00,802 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 495ms, sequenceid=415, compaction requested=false 2023-04-06 18:58:00,802 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:00,802 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:00,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:00,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:00,806 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:00,807 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35395781[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:00,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:00,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:00,810 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,810 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:00,810 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:00,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:00,813 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:00,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:00,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:00,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:00,813 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:00,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:00,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:00,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:00,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,818 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:00,818 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,918 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:00,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:00,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:00,926 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:00,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:00,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:00,926 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,927 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:00,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:00,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:00,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:00,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:00,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:00,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:00,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:00,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:00,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf7701e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:00,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:00,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:00,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:00,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:00,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:00,937 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:00,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:00,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:00,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:00,937 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:00,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:00,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:00,942 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:00,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:00,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:00,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:00,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,943 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:00,943 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:00,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:00,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,043 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:01,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:01,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:01,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:01,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:01,058 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:01,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:01,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:01,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,059 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:01,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:01,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:01,061 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,062 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:01,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:01,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:01,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:01,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,087 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:01,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@734eb90d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:01,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:01,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:01,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:01,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:01,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:01,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:01,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:01,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:01,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:01,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:01,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:01,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:01,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,113 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:01,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,114 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:01,114 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:01,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,214 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:01,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:01,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:01,226 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:01,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:01,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:01,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:01,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:01,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,236 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:01,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:01,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:01,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:01,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:01,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:01,270 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807481255, max timestamp=1680807481271Num uniques 859; 2023-04-06 18:58:01,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:01,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807481255, max timestamp=1680807481271Num uniques 859; 2023-04-06 18:58:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807481255, max timestamp=1680807481271Num uniques 859; 2023-04-06 18:58:01,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:01,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:01,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f301759[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:01,277 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:01,277 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,278 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:01,278 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:01,278 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:01,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:01,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,279 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:01,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:01,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:01,280 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:01,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:01,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:01,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:01,280 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:01,280 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,280 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:01,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:01,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:01,284 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,284 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:01,284 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,284 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:01,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,287 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406df452c5f9c564821b9aff21978c0505f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406df452c5f9c564821b9aff21978c0505f 2023-04-06 18:58:01,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807541298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807541299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,310 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/dc8ecc2a50fb459f9e0710c4c386b0d7 2023-04-06 18:58:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807541300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807541301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807541302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,333 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/422a6f4581bd4ee685aed387552af654 2023-04-06 18:58:01,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/ef95c27979964748986b21b98b95c3a7 2023-04-06 18:58:01,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/dc8ecc2a50fb459f9e0710c4c386b0d7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7 2023-04-06 18:58:01,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7, entries=350, sequenceid=429, filesize=52.9 K 2023-04-06 18:58:01,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/422a6f4581bd4ee685aed387552af654 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654 2023-04-06 18:58:01,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654, entries=150, sequenceid=429, filesize=11.9 K 2023-04-06 18:58:01,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/ef95c27979964748986b21b98b95c3a7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7 2023-04-06 18:58:01,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7, entries=150, sequenceid=429, filesize=11.9 K 2023-04-06 18:58:01,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 139ms, sequenceid=429, compaction requested=true 2023-04-06 18:58:01,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:01,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:58:01,370 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-06 18:58:01,371 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:58:01,371 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:58:01,371 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. because compaction request was cancelled 2023-04-06 18:58:01,371 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:01,372 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:01,372 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:58:01,372 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,372 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/fb118d13658547e3a6d726cb4459c72c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.4 K 2023-04-06 18:58:01,372 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting fb118d13658547e3a6d726cb4459c72c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:01,372 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting cf8656350ec64623a2f5ff4fe57d874b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1680807479018 2023-04-06 18:58:01,372 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 422a6f4581bd4ee685aed387552af654, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680807481202 2023-04-06 18:58:01,381 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:01,385 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:01,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:01,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:01,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:01,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:01,390 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:01,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:01,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,391 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/a619c46574cd49c381a4f9906de0f789 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/a619c46574cd49c381a4f9906de0f789 2023-04-06 18:58:01,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:01,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:01,396 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into a619c46574cd49c381a4f9906de0f789(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:01,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,396 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807481370; duration=0sec 2023-04-06 18:58:01,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:01,396 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:01,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:01,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:01,397 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:01,397 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:01,397 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:01,398 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:01,398 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:58:01,398 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,399 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2adf52cb7c44455f9178a6aa74d0d219, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.4 K 2023-04-06 18:58:01,399 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2adf52cb7c44455f9178a6aa74d0d219, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:01,399 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 655b6083aaf84f1a873a4c359542e82e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1680807479018 2023-04-06 18:58:01,400 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting ef95c27979964748986b21b98b95c3a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680807481202 2023-04-06 18:58:01,406 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:01,406 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,406 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:01,406 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:01,406 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-06 18:58:01,406 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:01,406 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:01,407 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:01,407 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:01,407 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:01,407 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:01,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:01,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:01,427 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:01,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807541431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807541431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,448 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/d084c71239c8494cb2d18fd1037877a2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d084c71239c8494cb2d18fd1037877a2 2023-04-06 18:58:01,450 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040632f49f1c913c4ca6b389f21000cd269c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040632f49f1c913c4ca6b389f21000cd269c 2023-04-06 18:58:01,453 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into d084c71239c8494cb2d18fd1037877a2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:01,453 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,453 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807481370; duration=0sec 2023-04-06 18:58:01,453 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:01,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807541446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807541446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807541447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,479 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/b3b982d511cc4563b963db950e057f3c 2023-04-06 18:58:01,516 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/10287c5cc18c47a98bfb460e1de2ac87 2023-04-06 18:58:01,534 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/fb5e5c10def64b3aa6323fc748e6edd3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/fb5e5c10def64b3aa6323fc748e6edd3 2023-04-06 18:58:01,542 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into fb5e5c10def64b3aa6323fc748e6edd3(size=26.2 K), total size for store is 104.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 18:58:01,542 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,542 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=13, startTime=1680807480263; duration=1sec 2023-04-06 18:58:01,542 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:01,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807541547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807541547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807541560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807541561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807541561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807541749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807541760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807541767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807541768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807541768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,961 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/92f56a18692b485e89d13c52b94569df 2023-04-06 18:58:01,964 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/b3b982d511cc4563b963db950e057f3c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c 2023-04-06 18:58:01,967 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c, entries=150, sequenceid=465, filesize=25.4 K 2023-04-06 18:58:01,968 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/10287c5cc18c47a98bfb460e1de2ac87 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87 2023-04-06 18:58:01,971 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87, entries=150, sequenceid=465, filesize=11.9 K 2023-04-06 18:58:01,972 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/92f56a18692b485e89d13c52b94569df as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df 2023-04-06 18:58:01,983 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df, entries=150, sequenceid=465, filesize=11.9 K 2023-04-06 18:58:01,984 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 577ms, sequenceid=465, compaction requested=true 2023-04-06 18:58:01,984 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:01,984 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:01,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:01,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:01,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:01,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:01,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:01,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@704adc61[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:01,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:01,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:01,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:01,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:01,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:01,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:01,991 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:01,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:01,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:01,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:01,991 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:01,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:01,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:01,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:01,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:01,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:01,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:01,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:01,998 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:01,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:01,998 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:01,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:01,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:01,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,106 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:02,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:58:02,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:02,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:02,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:02,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:02,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:02,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:02,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,112 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,113 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,119 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bd25f6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,133 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,136 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406aebd15d0680842888de58472e65bc774 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406aebd15d0680842888de58472e65bc774 2023-04-06 18:58:02,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,139 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,139 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,141 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,144 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,144 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807542142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807542144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807542144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807542150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807542151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,244 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,256 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,258 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807542252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807542253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807542255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807542260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807542260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bd6c3d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,266 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,266 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,272 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,274 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,274 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,278 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,278 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,379 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,381 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,381 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,381 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,381 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,382 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,383 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,384 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,384 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,384 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,384 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,384 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,385 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7938376c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,386 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,387 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,387 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,387 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,389 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,389 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,395 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,395 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807542460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807542461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807542461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807542462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807542463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,496 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,498 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,498 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,498 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,499 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,500 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,501 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,502 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb745fe[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,503 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,504 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,506 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,507 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,511 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,512 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,512 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,548 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=479, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/1316bc4e742242aca1751563e760fe68 2023-04-06 18:58:02,612 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,615 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,618 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,618 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,619 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463c496b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,620 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,621 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,621 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,623 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,624 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,638 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,638 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,739 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,741 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,741 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,747 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,755 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7d1915[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,758 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,763 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,763 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43016 deadline: 1680807542766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43056 deadline: 1680807542767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43040 deadline: 1680807542767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,772 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,772 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42990 deadline: 1680807542768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:02,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42968 deadline: 1680807542768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,774 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,775 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,775 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,775 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,775 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,780 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,786 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,790 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,789 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,790 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,790 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,890 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:02,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:02,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:02,893 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:02,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:02,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,894 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:02,897 DEBUG [Thread-2284] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x228ff1a4 to 127.0.0.1:63695 2023-04-06 18:58:02,897 DEBUG [Thread-2284] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:02,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:02,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:02,904 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:02,904 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:02,904 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:02,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:02,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:02,905 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,905 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. started... 2023-04-06 18:58:02,905 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. as already flushing 2023-04-06 18:58:02,905 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:02,905 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:02,906 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,906 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:02,906 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:02,908 DEBUG [Thread-2286] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x143a1290 to 127.0.0.1:63695 2023-04-06 18:58:02,908 DEBUG [Thread-2286] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:02,910 DEBUG [Thread-2280] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ded66ad to 127.0.0.1:63695 2023-04-06 18:58:02,910 DEBUG [Thread-2280] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:02,911 DEBUG [Thread-2288] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e613a44 to 127.0.0.1:63695 2023-04-06 18:58:02,911 DEBUG [Thread-2288] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:02,915 DEBUG [Thread-2282] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d061d2 to 127.0.0.1:63695 2023-04-06 18:58:02,915 DEBUG [Thread-2282] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:02,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,922 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,922 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:02,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6346d641[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:02,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:02,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:02,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:02,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:02,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:02,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:02,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:02,927 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:02,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:02,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:02,927 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:02,927 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:02,928 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:02,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:02,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:02,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:02,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:02,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:02,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:02,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:02,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:02,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:02,931 DEBUG [Thread-2279] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:02,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:02,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:02,931 DEBUG [Thread-2279] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:02,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:02,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:02,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:02,956 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6ead6c6fbe274d4580177632884fe470 2023-04-06 18:58:02,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=479 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/29ea1ca59fa344349821499fdba5b9c5 2023-04-06 18:58:02,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/1316bc4e742242aca1751563e760fe68 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68 2023-04-06 18:58:02,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68, entries=150, sequenceid=479, filesize=25.4 K 2023-04-06 18:58:02,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/6ead6c6fbe274d4580177632884fe470 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470 2023-04-06 18:58:02,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470, entries=150, sequenceid=479, filesize=11.9 K 2023-04-06 18:58:02,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/29ea1ca59fa344349821499fdba5b9c5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5 2023-04-06 18:58:02,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5, entries=150, sequenceid=479, filesize=11.9 K 2023-04-06 18:58:02,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 883ms, sequenceid=479, compaction requested=true 2023-04-06 18:58:02,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:02,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:02,992 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:02,992 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:02,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:02,993 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:02,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:02,993 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/B is initiating minor compaction (all files) 2023-04-06 18:58:02,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/B in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,993 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/a619c46574cd49c381a4f9906de0f789, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.5 K 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a619c46574cd49c381a4f9906de0f789, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680807481202 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159148 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/A is initiating minor compaction (all files) 2023-04-06 18:58:02,994 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/A in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 10287c5cc18c47a98bfb460e1de2ac87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680807481296 2023-04-06 18:58:02,994 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/fb5e5c10def64b3aa6323fc748e6edd3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=155.4 K 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6ead6c6fbe274d4580177632884fe470, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680807481428 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting fb5e5c10def64b3aa6323fc748e6edd3, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1680807478374 2023-04-06 18:58:02,994 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 460e7b538e834b9d90e1ae56d42fb199, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1680807479018 2023-04-06 18:58:02,995 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting dc8ecc2a50fb459f9e0710c4c386b0d7, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680807481126 2023-04-06 18:58:02,995 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b3b982d511cc4563b963db950e057f3c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680807481296 2023-04-06 18:58:02,995 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 1316bc4e742242aca1751563e760fe68, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680807481428 2023-04-06 18:58:03,007 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#B#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:03,027 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#A#compaction#377 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:03,031 DEBUG [Thread-2279] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:03,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:03,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:03,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-06 18:58:03,274 DEBUG [Thread-2271] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:63695 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:03,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:03,274 DEBUG [Thread-2271] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,276 DEBUG [Thread-2277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52249391 to 127.0.0.1:63695 2023-04-06 18:58:03,276 DEBUG [Thread-2277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,276 DEBUG [Thread-2273] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:63695 2023-04-06 18:58:03,276 DEBUG [Thread-2273] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,284 DEBUG [Thread-2275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:63695 2023-04-06 18:58:03,284 DEBUG [Thread-2275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,284 DEBUG [Thread-2269] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57b6628e to 127.0.0.1:63695 2023-04-06 18:58:03,284 DEBUG [Thread-2269] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 72 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1991 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5973 rows 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1979 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5936 rows 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1965 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5892 rows 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1970 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5908 rows 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 1976 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 5928 rows 2023-04-06 18:58:03,285 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:58:03,285 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74a355de to 127.0.0.1:63695 2023-04-06 18:58:03,286 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:03,286 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:58:03,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:58:03,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:03,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:03,291 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807483291"}]},"ts":"1680807483291"} 2023-04-06 18:58:03,291 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:58:03,378 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:58:03,378 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, UNASSIGN}] 2023-04-06 18:58:03,379 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, UNASSIGN 2023-04-06 18:58:03,379 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:03,379 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807483379"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807483379"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807483379"}]},"ts":"1680807483379"} 2023-04-06 18:58:03,380 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:03,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:03,422 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/07ce4aa80ec2494cb98741e8799e91ce as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/07ce4aa80ec2494cb98741e8799e91ce 2023-04-06 18:58:03,425 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/B of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 07ce4aa80ec2494cb98741e8799e91ce(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:03,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:03,425 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/B, priority=13, startTime=1680807482992; duration=0sec 2023-04-06 18:58:03,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:03,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:03,426 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:03,426 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 08d5d1fdb4f1757f6fcf6168ecd0a421/C is initiating minor compaction (all files) 2023-04-06 18:58:03,426 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 08d5d1fdb4f1757f6fcf6168ecd0a421/C in TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:03,426 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d084c71239c8494cb2d18fd1037877a2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp, totalSize=36.5 K 2023-04-06 18:58:03,427 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d084c71239c8494cb2d18fd1037877a2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680807481202 2023-04-06 18:58:03,427 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 92f56a18692b485e89d13c52b94569df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680807481296 2023-04-06 18:58:03,427 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 29ea1ca59fa344349821499fdba5b9c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=479, earliestPutTs=1680807481428 2023-04-06 18:58:03,432 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 08d5d1fdb4f1757f6fcf6168ecd0a421#C#compaction#379 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:03,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:03,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 08d5d1fdb4f1757f6fcf6168ecd0a421, disabling compactions & flushes 2023-04-06 18:58:03,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 2 compactions & cache flush to complete for region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:03,685 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069614398c519a4042ac66c3db6cb29f37 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069614398c519a4042ac66c3db6cb29f37 2023-04-06 18:58:03,690 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/6f2bdf0f17e04d7fa80617e16ed396a7 2023-04-06 18:58:03,840 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/57f1ab001b284fa8b9cdfd620434ae00 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/57f1ab001b284fa8b9cdfd620434ae00 2023-04-06 18:58:03,844 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/a7f19b0c4d8d4b4e9e5dcef7258da57c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a7f19b0c4d8d4b4e9e5dcef7258da57c 2023-04-06 18:58:03,844 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/C of 08d5d1fdb4f1757f6fcf6168ecd0a421 into 57f1ab001b284fa8b9cdfd620434ae00(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:03,844 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:03,844 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/C, priority=13, startTime=1680807482993; duration=0sec 2023-04-06 18:58:03,844 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:03,848 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 08d5d1fdb4f1757f6fcf6168ecd0a421/A of 08d5d1fdb4f1757f6fcf6168ecd0a421 into a7f19b0c4d8d4b4e9e5dcef7258da57c(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:03,848 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:03,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:03,848 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421., storeName=08d5d1fdb4f1757f6fcf6168ecd0a421/A, priority=11, startTime=1680807482992; duration=0sec 2023-04-06 18:58:03,848 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:03,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:04,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/e0a43fe506c74500a3afbed9291eca42 2023-04-06 18:58:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:04,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/835e688d366a4835a1c9e049046b0c2b 2023-04-06 18:58:04,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/6f2bdf0f17e04d7fa80617e16ed396a7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/6f2bdf0f17e04d7fa80617e16ed396a7 2023-04-06 18:58:04,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/6f2bdf0f17e04d7fa80617e16ed396a7, entries=150, sequenceid=500, filesize=25.4 K 2023-04-06 18:58:04,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/e0a43fe506c74500a3afbed9291eca42 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0a43fe506c74500a3afbed9291eca42 2023-04-06 18:58:04,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0a43fe506c74500a3afbed9291eca42, entries=150, sequenceid=500, filesize=11.9 K 2023-04-06 18:58:04,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/835e688d366a4835a1c9e049046b0c2b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/835e688d366a4835a1c9e049046b0c2b 2023-04-06 18:58:04,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/835e688d366a4835a1c9e049046b0c2b, entries=150, sequenceid=500, filesize=11.9 K 2023-04-06 18:58:04,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1252ms, sequenceid=500, compaction requested=false 2023-04-06 18:58:04,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:04,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:04,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:04,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. after waiting 0 ms 2023-04-06 18:58:04,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:04,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 08d5d1fdb4f1757f6fcf6168ecd0a421 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:58:04,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=A 2023-04-06 18:58:04,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=B 2023-04-06 18:58:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 08d5d1fdb4f1757f6fcf6168ecd0a421, store=C 2023-04-06 18:58:04,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:04,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304068ef2fd895ecc4d0dba8cf25abb685884 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068ef2fd895ecc4d0dba8cf25abb685884 2023-04-06 18:58:05,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/85a5c78904414aa58f16acd3a421aca4 2023-04-06 18:58:05,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:05,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/062828254b214fe59ddfd38249c96e34 2023-04-06 18:58:06,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/5994b9ad242f4553aee67c1b8e274c29 2023-04-06 18:58:06,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/A/85a5c78904414aa58f16acd3a421aca4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/85a5c78904414aa58f16acd3a421aca4 2023-04-06 18:58:06,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/85a5c78904414aa58f16acd3a421aca4, entries=150, sequenceid=510, filesize=25.4 K 2023-04-06 18:58:06,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/B/062828254b214fe59ddfd38249c96e34 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/062828254b214fe59ddfd38249c96e34 2023-04-06 18:58:06,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/062828254b214fe59ddfd38249c96e34, entries=150, sequenceid=510, filesize=11.9 K 2023-04-06 18:58:06,175 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/.tmp/C/5994b9ad242f4553aee67c1b8e274c29 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/5994b9ad242f4553aee67c1b8e274c29 2023-04-06 18:58:06,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/5994b9ad242f4553aee67c1b8e274c29, entries=150, sequenceid=510, filesize=11.9 K 2023-04-06 18:58:06,178 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 08d5d1fdb4f1757f6fcf6168ecd0a421 in 1652ms, sequenceid=510, compaction requested=true 2023-04-06 18:58:06,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e6ce018744a643b39b90ccec9e7bd0da, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ccb64d5770e94f728153953dc065f0e8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dddc54241f9e4e61ba46e4a1e742cb79, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a208d8a8a4344d1da7a6fa3d7d003c67, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/157c6269ecee4db8a3b9dc71950f30f4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e37ea8bacf404f3788b6184f3b9366db, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/fb5e5c10def64b3aa6323fc748e6edd3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68] to archive 2023-04-06 18:58:06,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:06,204 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/f75edc0776b442ff811f68cb71f03424 2023-04-06 18:58:06,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/0a8c7c86cad3456ab3f010d8e0c48dbd 2023-04-06 18:58:06,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/58e963fa17cb44e9a865cb4d4d520524 2023-04-06 18:58:06,207 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e6ce018744a643b39b90ccec9e7bd0da to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e6ce018744a643b39b90ccec9e7bd0da 2023-04-06 18:58:06,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/347c8c049679450d9f318d31b16a4e9c 2023-04-06 18:58:06,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/303e457ceba84aacacffac1d74338342 2023-04-06 18:58:06,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ccb64d5770e94f728153953dc065f0e8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ccb64d5770e94f728153953dc065f0e8 2023-04-06 18:58:06,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c75f4590ef8048d093ceeca75bbbf81d 2023-04-06 18:58:06,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e4774d2f70924f50a1519a980a9ab2dc 2023-04-06 18:58:06,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dddc54241f9e4e61ba46e4a1e742cb79 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dddc54241f9e4e61ba46e4a1e742cb79 2023-04-06 18:58:06,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/290802d3dc5b4c50870e6494a51668c5 2023-04-06 18:58:06,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/80c0795605bd4a4fb490e1f595678c08 2023-04-06 18:58:06,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a208d8a8a4344d1da7a6fa3d7d003c67 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a208d8a8a4344d1da7a6fa3d7d003c67 2023-04-06 18:58:06,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/9f9e20a614e3430dad61c01569b677b8 2023-04-06 18:58:06,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dab47d52b3e7410d9608bae21681bbfb 2023-04-06 18:58:06,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/ae323afaa865442bb62e6f61d6ccd5fa 2023-04-06 18:58:06,216 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/157c6269ecee4db8a3b9dc71950f30f4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/157c6269ecee4db8a3b9dc71950f30f4 2023-04-06 18:58:06,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a589c3be184b49e9b99b11fc1287e251 2023-04-06 18:58:06,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e37ea8bacf404f3788b6184f3b9366db to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/e37ea8bacf404f3788b6184f3b9366db 2023-04-06 18:58:06,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7cf2ae44ae704831b61aafc85ca0e74b 2023-04-06 18:58:06,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/c1ea5f43c2de4728bdc9c108e747b6a3 2023-04-06 18:58:06,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/7ce0800a7ccf4b6ba17165682e3564db 2023-04-06 18:58:06,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/fb5e5c10def64b3aa6323fc748e6edd3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/fb5e5c10def64b3aa6323fc748e6edd3 2023-04-06 18:58:06,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/460e7b538e834b9d90e1ae56d42fb199 2023-04-06 18:58:06,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/dc8ecc2a50fb459f9e0710c4c386b0d7 2023-04-06 18:58:06,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/b3b982d511cc4563b963db950e057f3c 2023-04-06 18:58:06,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/1316bc4e742242aca1751563e760fe68 2023-04-06 18:58:06,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f32de11d8ee24e4497bbc8483a98a33f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/7951c28e93844b5db8cfe3146bd02ff7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/37020b9fcc634b1ba5750aea4fbc8898, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0279911ed3746988983fdb252b76207, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/df1f3c05ad8543d1935891ea5bd5f201, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/9b70839baed24b9ab9751eb441b90ee4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/fb118d13658547e3a6d726cb4459c72c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/a619c46574cd49c381a4f9906de0f789, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470] to archive 2023-04-06 18:58:06,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:06,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/384b9753b3bc4077af8f594c24a58b0f 2023-04-06 18:58:06,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/05894bce47e64a5ba24daf8bbab59f35 2023-04-06 18:58:06,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f32de11d8ee24e4497bbc8483a98a33f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f32de11d8ee24e4497bbc8483a98a33f 2023-04-06 18:58:06,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/856b6bbe4719435b84b20e10bae409da 2023-04-06 18:58:06,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/f29f15d0a2854c5da6576326c9965a48 2023-04-06 18:58:06,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/7951c28e93844b5db8cfe3146bd02ff7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/7951c28e93844b5db8cfe3146bd02ff7 2023-04-06 18:58:06,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/04b859bb336e4881b3fb87acc9d61443 2023-04-06 18:58:06,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/d88ce34df7cf4b85a4a21386052af760 2023-04-06 18:58:06,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/37020b9fcc634b1ba5750aea4fbc8898 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/37020b9fcc634b1ba5750aea4fbc8898 2023-04-06 18:58:06,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b6bc660855d34b109a84a1f7d133b1c8 2023-04-06 18:58:06,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/1fd6e6fae1df49368c4375d7cdfdc5bd 2023-04-06 18:58:06,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0279911ed3746988983fdb252b76207 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0279911ed3746988983fdb252b76207 2023-04-06 18:58:06,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/39206994325b42d2b47d2dd49ca893e0 2023-04-06 18:58:06,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6d9823f3d86f4894a9393cccc7eae034 2023-04-06 18:58:06,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6157941dc7e8412eb0399108bd9f7024 2023-04-06 18:58:06,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/df1f3c05ad8543d1935891ea5bd5f201 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/df1f3c05ad8543d1935891ea5bd5f201 2023-04-06 18:58:06,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/b524fa9af3bc44a3b3a1d6dde9b6c838 2023-04-06 18:58:06,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/78683d2ac8354a568bc5cf581c0e5999 2023-04-06 18:58:06,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/9b70839baed24b9ab9751eb441b90ee4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/9b70839baed24b9ab9751eb441b90ee4 2023-04-06 18:58:06,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/019b61d1644a423d87300b444547fba2 2023-04-06 18:58:06,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/dbbafc18007f4589b177435e4c16d283 2023-04-06 18:58:06,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/fb118d13658547e3a6d726cb4459c72c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/fb118d13658547e3a6d726cb4459c72c 2023-04-06 18:58:06,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/869e6e7250454ecb9188063cc4ba9858 2023-04-06 18:58:06,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/cf8656350ec64623a2f5ff4fe57d874b 2023-04-06 18:58:06,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/a619c46574cd49c381a4f9906de0f789 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/a619c46574cd49c381a4f9906de0f789 2023-04-06 18:58:06,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/422a6f4581bd4ee685aed387552af654 2023-04-06 18:58:06,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/10287c5cc18c47a98bfb460e1de2ac87 2023-04-06 18:58:06,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/6ead6c6fbe274d4580177632884fe470 2023-04-06 18:58:06,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a426fc4f74444f19afd34a220090da2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/913fc6f2a6914f9cbfb7764e57baed2e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/b8edec4f33084389945efa798e6aafe5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/715a9c9553ea420c9261127eec13834b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1c9fd57678c14d009949a9dd95440f9d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/393c3ad815754ceba01105833f5170d2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2adf52cb7c44455f9178a6aa74d0d219, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d084c71239c8494cb2d18fd1037877a2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5] to archive 2023-04-06 18:58:06,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:06,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2a9d59497c1f4f169fb5e75a68fa278f 2023-04-06 18:58:06,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92bddfdd8a9a4dec9576e0750ab7638e 2023-04-06 18:58:06,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a426fc4f74444f19afd34a220090da2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a426fc4f74444f19afd34a220090da2 2023-04-06 18:58:06,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e434c2ec893940bf9330b733d7445925 2023-04-06 18:58:06,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/25c6b56f573d490e830aa1163f2a3894 2023-04-06 18:58:06,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/913fc6f2a6914f9cbfb7764e57baed2e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/913fc6f2a6914f9cbfb7764e57baed2e 2023-04-06 18:58:06,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/03d8f298de184be8832e71968b3955ae 2023-04-06 18:58:06,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/8a6e6a63216641eb96a3681b1bb12427 2023-04-06 18:58:06,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/b8edec4f33084389945efa798e6aafe5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/b8edec4f33084389945efa798e6aafe5 2023-04-06 18:58:06,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e554d36637f14da5856d7c689a5260b3 2023-04-06 18:58:06,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c34f53315fc24a119485bb582326a3f9 2023-04-06 18:58:06,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/715a9c9553ea420c9261127eec13834b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/715a9c9553ea420c9261127eec13834b 2023-04-06 18:58:06,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/293279b8e29b40758f9257e2465f6adb 2023-04-06 18:58:06,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/c55e1b1fe08946468c92e8e2da9f78ff 2023-04-06 18:58:06,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1653f7a7e39d4947aad49598c97244fe 2023-04-06 18:58:06,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1c9fd57678c14d009949a9dd95440f9d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/1c9fd57678c14d009949a9dd95440f9d 2023-04-06 18:58:06,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/f5f0e242962c487eb717f6ed00810b70 2023-04-06 18:58:06,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/de3fcc939d014b569a510c28f13be872 2023-04-06 18:58:06,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/393c3ad815754ceba01105833f5170d2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/393c3ad815754ceba01105833f5170d2 2023-04-06 18:58:06,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d83afd17fbbe4e64bc55c0d97907cb30 2023-04-06 18:58:06,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/e455968589e34c9688be62e47b48dc02 2023-04-06 18:58:06,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2adf52cb7c44455f9178a6aa74d0d219 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/2adf52cb7c44455f9178a6aa74d0d219 2023-04-06 18:58:06,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/9b98dc5fb8d74645a20167cab797c7e8 2023-04-06 18:58:06,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/655b6083aaf84f1a873a4c359542e82e 2023-04-06 18:58:06,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d084c71239c8494cb2d18fd1037877a2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/d084c71239c8494cb2d18fd1037877a2 2023-04-06 18:58:06,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/ef95c27979964748986b21b98b95c3a7 2023-04-06 18:58:06,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/92f56a18692b485e89d13c52b94569df 2023-04-06 18:58:06,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/29ea1ca59fa344349821499fdba5b9c5 2023-04-06 18:58:06,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits/513.seqid, newMaxSeqId=513, maxSeqId=4 2023-04-06 18:58:06,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421. 2023-04-06 18:58:06,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 08d5d1fdb4f1757f6fcf6168ecd0a421: 2023-04-06 18:58:06,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:06,274 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=08d5d1fdb4f1757f6fcf6168ecd0a421, regionState=CLOSED 2023-04-06 18:58:06,274 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807486274"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807486274"}]},"ts":"1680807486274"} 2023-04-06 18:58:06,276 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-06 18:58:06,276 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 08d5d1fdb4f1757f6fcf6168ecd0a421, server=jenkins-hbase20.apache.org,34607,1680807373848 in 2.8950 sec 2023-04-06 18:58:06,277 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-06 18:58:06,277 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=08d5d1fdb4f1757f6fcf6168ecd0a421, UNASSIGN in 2.8980 sec 2023-04-06 18:58:06,277 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807486277"}]},"ts":"1680807486277"} 2023-04-06 18:58:06,278 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:58:06,279 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:58:06,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9920 sec 2023-04-06 18:58:07,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 18:58:07,396 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-06 18:58:07,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:58:07,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,398 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,399 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 18:58:07,403 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:07,405 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits] 2023-04-06 18:58:07,407 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/6f2bdf0f17e04d7fa80617e16ed396a7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/6f2bdf0f17e04d7fa80617e16ed396a7 2023-04-06 18:58:07,408 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/85a5c78904414aa58f16acd3a421aca4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/85a5c78904414aa58f16acd3a421aca4 2023-04-06 18:58:07,409 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a7f19b0c4d8d4b4e9e5dcef7258da57c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/A/a7f19b0c4d8d4b4e9e5dcef7258da57c 2023-04-06 18:58:07,411 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/062828254b214fe59ddfd38249c96e34 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/062828254b214fe59ddfd38249c96e34 2023-04-06 18:58:07,412 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/07ce4aa80ec2494cb98741e8799e91ce to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/07ce4aa80ec2494cb98741e8799e91ce 2023-04-06 18:58:07,412 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0a43fe506c74500a3afbed9291eca42 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/B/e0a43fe506c74500a3afbed9291eca42 2023-04-06 18:58:07,414 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/57f1ab001b284fa8b9cdfd620434ae00 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/57f1ab001b284fa8b9cdfd620434ae00 2023-04-06 18:58:07,415 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/5994b9ad242f4553aee67c1b8e274c29 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/5994b9ad242f4553aee67c1b8e274c29 2023-04-06 18:58:07,416 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/835e688d366a4835a1c9e049046b0c2b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/C/835e688d366a4835a1c9e049046b0c2b 2023-04-06 18:58:07,418 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits/513.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421/recovered.edits/513.seqid 2023-04-06 18:58:07,418 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/08d5d1fdb4f1757f6fcf6168ecd0a421 2023-04-06 18:58:07,418 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:58:07,418 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:58:07,419 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 18:58:07,421 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406265419c0eb5c48b8b33fefeb41b8a18e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406265419c0eb5c48b8b33fefeb41b8a18e 2023-04-06 18:58:07,422 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040632f49f1c913c4ca6b389f21000cd269c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040632f49f1c913c4ca6b389f21000cd269c 2023-04-06 18:58:07,423 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406615f52b7ebe640e6a7db59a9e71370d7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406615f52b7ebe640e6a7db59a9e71370d7 2023-04-06 18:58:07,423 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066ef798fac08d4f1c92509875a8bc3f35 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066ef798fac08d4f1c92509875a8bc3f35 2023-04-06 18:58:07,424 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406736acee33d16465c9ecfef841e8c4eda to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406736acee33d16465c9ecfef841e8c4eda 2023-04-06 18:58:07,425 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040674895a9d832d4927aeacdb5067c8c7b3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040674895a9d832d4927aeacdb5067c8c7b3 2023-04-06 18:58:07,425 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040675c31659f7f649bc910e256502a71f6d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040675c31659f7f649bc910e256502a71f6d 2023-04-06 18:58:07,426 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406881c241df6c5413ab299e1bf63d1e109 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406881c241df6c5413ab299e1bf63d1e109 2023-04-06 18:58:07,427 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068c5df0c5adec42759c0c69a6d336564a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068c5df0c5adec42759c0c69a6d336564a 2023-04-06 18:58:07,428 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068ef2fd895ecc4d0dba8cf25abb685884 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304068ef2fd895ecc4d0dba8cf25abb685884 2023-04-06 18:58:07,428 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069614398c519a4042ac66c3db6cb29f37 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069614398c519a4042ac66c3db6cb29f37 2023-04-06 18:58:07,429 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069fc5bdd70f6c4ed6bd3fd776849c0a70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069fc5bdd70f6c4ed6bd3fd776849c0a70 2023-04-06 18:58:07,430 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a63e1de0989a41c9bf26787dcb7099e0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a63e1de0989a41c9bf26787dcb7099e0 2023-04-06 18:58:07,431 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406aebd15d0680842888de58472e65bc774 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406aebd15d0680842888de58472e65bc774 2023-04-06 18:58:07,432 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b2dcccd1666d4d6597b4baeacbef5595 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b2dcccd1666d4d6597b4baeacbef5595 2023-04-06 18:58:07,433 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b549a7cb73a448dea1c3bdc635e70562 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b549a7cb73a448dea1c3bdc635e70562 2023-04-06 18:58:07,433 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6efd7b1706244e2aee6365374d40985 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b6efd7b1706244e2aee6365374d40985 2023-04-06 18:58:07,434 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b890a077d5214956828d89b1ffeca60d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b890a077d5214956828d89b1ffeca60d 2023-04-06 18:58:07,435 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ca877adf58cd44068fe41cdec77073b7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ca877adf58cd44068fe41cdec77073b7 2023-04-06 18:58:07,436 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d52bf765bbc4493aa1bbf00675d8fbc4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d52bf765bbc4493aa1bbf00675d8fbc4 2023-04-06 18:58:07,437 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d6d3a1010f9c46b6a9c5b865a974ecf5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d6d3a1010f9c46b6a9c5b865a974ecf5 2023-04-06 18:58:07,438 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406df452c5f9c564821b9aff21978c0505f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406df452c5f9c564821b9aff21978c0505f 2023-04-06 18:58:07,438 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:58:07,440 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,441 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:58:07,442 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:58:07,443 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,443 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:58:07,444 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807487443"}]},"ts":"9223372036854775807"} 2023-04-06 18:58:07,445 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:58:07,445 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 08d5d1fdb4f1757f6fcf6168ecd0a421, NAME => 'TestAcidGuarantees,,1680807459747.08d5d1fdb4f1757f6fcf6168ecd0a421.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:58:07,445 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:58:07,445 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807487445"}]},"ts":"9223372036854775807"} 2023-04-06 18:58:07,447 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:58:07,448 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,449 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-04-06 18:58:07,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 18:58:07,500 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-06 18:58:07,510 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 229), OpenFileDescriptor=583 (was 554) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=493 (was 527), ProcessCount=166 (was 166), AvailableMemoryMB=4185 (was 2854) - AvailableMemoryMB LEAK? - 2023-04-06 18:58:07,519 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=493, ProcessCount=166, AvailableMemoryMB=4184 2023-04-06 18:58:07,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:58:07,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:58:07,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:07,524 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:58:07,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-06 18:58:07,524 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:58:07,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:07,525 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:07,526 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc empty. 2023-04-06 18:58:07,526 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:07,526 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:58:07,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:07,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:07,933 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:58:07,934 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 50e5e6c3a16e85ee7fcdf95128e439dc, NAME => 'TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:58:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:08,343 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:58:08,344 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 50e5e6c3a16e85ee7fcdf95128e439dc, disabling compactions & flushes 2023-04-06 18:58:08,344 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,344 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,344 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. after waiting 0 ms 2023-04-06 18:58:08,344 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,344 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,344 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:08,346 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:58:08,346 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807488346"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807488346"}]},"ts":"1680807488346"} 2023-04-06 18:58:08,347 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:58:08,348 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:58:08,348 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807488348"}]},"ts":"1680807488348"} 2023-04-06 18:58:08,349 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:58:08,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, ASSIGN}] 2023-04-06 18:58:08,352 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, ASSIGN 2023-04-06 18:58:08,352 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:58:08,503 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=50e5e6c3a16e85ee7fcdf95128e439dc, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:08,503 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807488503"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807488503"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807488503"}]},"ts":"1680807488503"} 2023-04-06 18:58:08,504 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:08,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:08,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 50e5e6c3a16e85ee7fcdf95128e439dc, NAME => 'TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:58:08,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:58:08,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,661 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,662 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A 2023-04-06 18:58:08,662 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A 2023-04-06 18:58:08,663 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:08,663 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50e5e6c3a16e85ee7fcdf95128e439dc columnFamilyName A 2023-04-06 18:58:08,663 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(310): Store=50e5e6c3a16e85ee7fcdf95128e439dc/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:08,664 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,665 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B 2023-04-06 18:58:08,665 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B 2023-04-06 18:58:08,665 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:08,665 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50e5e6c3a16e85ee7fcdf95128e439dc columnFamilyName B 2023-04-06 18:58:08,666 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(310): Store=50e5e6c3a16e85ee7fcdf95128e439dc/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:08,666 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,667 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C 2023-04-06 18:58:08,667 DEBUG [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C 2023-04-06 18:58:08,667 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:08,667 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50e5e6c3a16e85ee7fcdf95128e439dc columnFamilyName C 2023-04-06 18:58:08,668 INFO [StoreOpener-50e5e6c3a16e85ee7fcdf95128e439dc-1] regionserver.HStore(310): Store=50e5e6c3a16e85ee7fcdf95128e439dc/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:08,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:58:08,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:08,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:58:08,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 50e5e6c3a16e85ee7fcdf95128e439dc; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70270600, jitterRate=0.04711353778839111}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:58:08,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:08,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., pid=49, masterSystemTime=1680807488656 2023-04-06 18:58:08,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:08,676 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=50e5e6c3a16e85ee7fcdf95128e439dc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:08,676 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807488676"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807488676"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807488676"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807488676"}]},"ts":"1680807488676"} 2023-04-06 18:58:08,678 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-06 18:58:08,678 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 in 173 msec 2023-04-06 18:58:08,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-06 18:58:08,679 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, ASSIGN in 328 msec 2023-04-06 18:58:08,680 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:58:08,680 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807488680"}]},"ts":"1680807488680"} 2023-04-06 18:58:08,681 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:58:08,682 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:58:08,683 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-06 18:58:09,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 18:58:09,630 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-06 18:58:09,630 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5a7ac4a1 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,639 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846250, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,640 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,642 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,643 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:58:09,645 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:58:09,646 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x682c0b9e to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,656 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@647f95f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,656 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x50d6899d to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,665 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58ae18e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,666 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6662e946 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,673 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29c1e76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,673 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5d7deb76 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,678 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c240146, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,679 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,683 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,683 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x6523d347 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,690 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@209faa3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,690 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x28072326 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,695 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5345748a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,695 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x2d90797d to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,701 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9f61fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,701 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7d3df389 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,713 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@534826c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,714 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x65f69c3f to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:09,731 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77459ab3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:09,763 DEBUG [hconnection-0x2fcaf61d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,763 DEBUG [hconnection-0x13fe8efb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,763 DEBUG [hconnection-0x2bf810dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,763 DEBUG [hconnection-0x3cf0cff8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,763 DEBUG [hconnection-0xf136cdb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:09,764 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,764 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,766 DEBUG [hconnection-0x58d8de28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,769 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,769 DEBUG [hconnection-0x175e0edc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,772 DEBUG [hconnection-0x4e383b5f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,772 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,773 DEBUG [hconnection-0xb2ac7d7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,774 DEBUG [hconnection-0x5e26cfdc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:09,773 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,778 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,780 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57206, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,784 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,784 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:09,787 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:09,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:09,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:09,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:09,791 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,792 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,794 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:09,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:09,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:09,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:09,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:09,797 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,798 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:09,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:09,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807549807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807549808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807549808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=509, cellSize=23305, totalHeapSize=53176, min timestamp=1680807489802, max timestamp=1680807489808Num uniques 859; 2023-04-06 18:58:09,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=509, cellSize=23305, totalHeapSize=53176, min timestamp=1680807489802, max timestamp=1680807489808Num uniques 859; 2023-04-06 18:58:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807549809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=23305, suffix heap size=117064,new segment heap size=53176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=509 2023-04-06 18:58:09,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=509, cellSize=23305, totalHeapSize=53176, min timestamp=1680807489802, max timestamp=1680807489808Num uniques 859; 2023-04-06 18:58:09,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807549809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=23305, suffix heap size=117064,new segment heap size=53176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=509 2023-04-06 18:58:09,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=23305, suffix heap size=117064,new segment heap size=53176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=509 2023-04-06 18:58:09,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:09,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:09,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:09,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:09,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:09,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:09,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:09,812 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:09,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:09,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:09,815 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63bf762[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:09,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:09,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:09,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:09,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:09,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:09,817 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:09,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:09,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:09,820 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:09,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:09,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:09,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:09,820 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:09,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:09,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,825 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:09,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:09,827 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:09,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:09,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:09,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:09,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:09,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,829 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:09,829 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:09,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807549914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807549915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807549916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807549917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807549917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,929 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:09,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:09,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:09,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:09,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:09,932 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:09,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:09,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:09,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:09,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:09,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:09,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,933 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:09,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:09,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:09,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:09,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:09,938 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:09,942 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:09,942 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:09,942 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:09,942 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:09,942 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:09,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:09,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e881df[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:09,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:09,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:09,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:09,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:09,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:09,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:09,960 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:09,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:09,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:09,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:09,960 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:09,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,961 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:09,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:09,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:09,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:09,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:09,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:09,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:09,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:09,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:09,965 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:09,965 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:09,965 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:09,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:09,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807550018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807550018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807550019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807550019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807550020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,065 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,074 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,075 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,082 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,088 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,089 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,089 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@116e8126[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,093 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,095 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,096 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,100 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,101 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,101 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,201 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,206 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,207 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,218 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,218 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,218 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,218 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/4bf2d8d73e1c4709a345a5dbebe39db7 2023-04-06 18:58:10,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cfa994[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,224 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,224 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,225 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,225 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,225 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,227 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,227 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,227 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807550226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,227 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807550226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,232 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,232 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,232 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,232 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,232 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807550227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807550227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807550227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,241 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,246 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,246 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,247 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,247 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,347 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,352 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,362 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d16c688[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,366 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,366 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,366 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,367 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,369 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,369 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,371 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,374 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,374 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,374 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,475 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,477 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,480 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,480 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,480 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,480 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,480 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,481 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,483 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,484 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b965a8c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,484 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,485 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,485 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,489 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,489 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,490 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,496 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,497 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,497 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807550528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807550528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807550541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807550541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807550546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,597 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,600 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,600 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,606 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,606 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,607 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,607 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,608 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e948369[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,608 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,609 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,609 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,609 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,609 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,609 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,610 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,611 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,612 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,612 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,612 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,613 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,613 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,616 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,616 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,616 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/7e9afbc384a348a0bfcec394bebd0738 2023-04-06 18:58:10,716 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,718 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,718 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,720 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,720 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,720 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,721 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,721 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,722 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,722 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,723 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272e0a26[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,725 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,725 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,725 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,725 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,726 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,727 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,728 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,729 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,729 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,733 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,733 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,733 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,734 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,734 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,734 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,834 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,837 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,838 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,839 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,839 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,839 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,839 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,839 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,840 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca7aacd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,845 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,845 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,845 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,847 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,849 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,849 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,849 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,949 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:10,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:10,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:10,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:10,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:10,952 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:10,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:10,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:10,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,953 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:10,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:10,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:10,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:10,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:10,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:10,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:10,955 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,955 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:10,955 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:10,955 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:10,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:10,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:10,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452cc3ef[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:10,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:10,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:10,958 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:10,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:10,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:10,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:10,961 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:10,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:10,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:10,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:10,961 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:10,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:10,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:10,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:10,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:10,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:10,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:10,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:10,965 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:10,964 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:10,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:10,965 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:10,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:10,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:10,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:10,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:10,965 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:11,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807551035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:11,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807551038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:11,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807551049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:11,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807551049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:11,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807551049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,065 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:11,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:11,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:11,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:11,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:11,068 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:11,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:11,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:11,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:11,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:11,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:11,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:11,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:11,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:11,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:11,070 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:11,070 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,071 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:11,071 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:11,071 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:11,071 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,072 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:11,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:11,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f198fe8[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:11,073 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,073 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:11,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,074 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:11,074 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:11,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:11,076 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:11,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:11,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:11,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:11,076 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:11,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/6d4f7a10a3a34ff69c0813cd9b2b0a57 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:11,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,081 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:11,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:11,081 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:11,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,081 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/4bf2d8d73e1c4709a345a5dbebe39db7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7 2023-04-06 18:58:11,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 18:58:11,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/7e9afbc384a348a0bfcec394bebd0738 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738 2023-04-06 18:58:11,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 18:58:11,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/6d4f7a10a3a34ff69c0813cd9b2b0a57 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57 2023-04-06 18:58:11,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57, entries=150, sequenceid=12, filesize=11.6 K 2023-04-06 18:58:11,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1305ms, sequenceid=12, compaction requested=false 2023-04-06 18:58:11,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:11,181 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:11,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:11,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:11,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:11,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:11,183 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:11,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:11,183 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:11,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,184 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,184 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,185 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:11,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:11,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,185 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:11,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:11,186 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:11,186 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=134.18 KB heapSize=352.17 KB 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:11,204 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=42 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/a1bc0c152be6483abf8bea4a70275c4e 2023-04-06 18:58:11,230 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=42 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/03a1898d700c433ca8596ae468cb9fff 2023-04-06 18:58:11,688 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=42 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/72f70a2b30bd4202953225d2aa560b55 2023-04-06 18:58:11,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/a1bc0c152be6483abf8bea4a70275c4e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e 2023-04-06 18:58:11,695 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e, entries=150, sequenceid=42, filesize=11.6 K 2023-04-06 18:58:11,696 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/03a1898d700c433ca8596ae468cb9fff as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff 2023-04-06 18:58:11,699 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff, entries=150, sequenceid=42, filesize=11.6 K 2023-04-06 18:58:11,699 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/72f70a2b30bd4202953225d2aa560b55 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55 2023-04-06 18:58:11,703 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55, entries=150, sequenceid=42, filesize=11.6 K 2023-04-06 18:58:11,703 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~352.13 KB/360576, currentSize=0 B/0 for 50e5e6c3a16e85ee7fcdf95128e439dc in 517ms, sequenceid=42, compaction requested=false 2023-04-06 18:58:11,703 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:11,703 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:11,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:11,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:11,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:11,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:11,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2831d517[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:11,707 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:11,707 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,708 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:11,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:11,708 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:11,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:11,709 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:11,709 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,709 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,709 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:11,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:11,712 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:11,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:11,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:11,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:11,712 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:11,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,719 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,720 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:11,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,720 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:11,720 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:11,821 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:11,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:11,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:11,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:11,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:11,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:11,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:11,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:11,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:11,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:11,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:11,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:11,829 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:11,830 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:58:11,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:11,830 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:11,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:11,831 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,832 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,832 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1971f31e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:11,833 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:11,833 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:11,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,835 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:11,837 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:11,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:11,839 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:11,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:11,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:11,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:11,839 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:11,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,841 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,844 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:11,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:11,844 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,844 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:11,844 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:11,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:11,945 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:11,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:11,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:11,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:11,948 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:11,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:11,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:11,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,950 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:11,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:11,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:11,951 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:11,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,951 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:11,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:11,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:11,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:11,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:11,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:11,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:11,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:11,954 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:11,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a617864[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:11,955 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:11,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:11,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:11,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:11,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:11,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:11,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:11,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:11,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:11,962 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:11,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:11,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:11,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:11,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:11,973 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:11,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:11,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:11,972 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:11,973 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:11,973 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:11,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:11,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:12,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:12,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:12,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:12,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:12,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:12,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:12,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:12,073 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,076 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,081 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,081 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,081 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,081 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,081 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,082 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@934d60f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,084 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,085 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,085 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,085 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,086 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,087 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,088 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,088 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,092 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,092 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,092 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,092 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807492097, max timestamp=1680807492107Num uniques 859; 2023-04-06 18:58:12,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807492097, max timestamp=1680807492107Num uniques 859; 2023-04-06 18:58:12,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:12,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:12,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807492097, max timestamp=1680807492107Num uniques 859; 2023-04-06 18:58:12,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:12,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807552142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807552143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807552144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807552144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807552151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,192 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,195 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,195 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,195 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,195 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,195 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,195 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,196 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,197 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,197 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,197 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,197 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,197 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,198 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8b2353[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,200 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,200 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,200 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,200 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,201 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,202 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,202 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,202 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,202 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,202 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,203 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,203 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,203 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,203 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,222 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,222 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807552252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807552252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807552254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807552255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807552257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,322 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,325 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,325 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,327 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,327 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,327 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,327 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,327 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f4ccbf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,330 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,330 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,330 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,331 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,333 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,333 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,337 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,337 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,337 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,337 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,437 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,442 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,443 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,443 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,446 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,447 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,447 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,447 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,448 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,448 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86721bd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,450 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,453 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,453 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,454 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,454 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,454 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,457 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,457 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807552459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807552459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807552460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807552460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/463c45516fde4fb6baccad6f7fccd584 2023-04-06 18:58:12,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807552466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,557 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,561 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,562 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,563 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274c1a10[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,568 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,568 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,570 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,572 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,572 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,573 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,573 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,573 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,576 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,577 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,577 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,577 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,677 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,679 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,680 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51ab3411[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,689 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,689 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,691 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,693 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,694 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,694 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,695 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,695 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807552765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807552765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807552766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807552769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:12,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807552774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,795 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,799 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,807 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,807 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,808 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,808 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e259baf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,816 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,819 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,829 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,829 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,830 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,830 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,830 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,833 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,839 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,839 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:12,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:12,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/0f59ddee6a7a4b58b53806b8adb0de89 2023-04-06 18:58:12,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/1660e73776434f2fba8b7595905c5a73 2023-04-06 18:58:12,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/463c45516fde4fb6baccad6f7fccd584 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584 2023-04-06 18:58:12,939 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:12,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:12,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:12,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584, entries=250, sequenceid=54, filesize=16.2 K 2023-04-06 18:58:12,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/0f59ddee6a7a4b58b53806b8adb0de89 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89 2023-04-06 18:58:12,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:12,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:12,944 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:12,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:12,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:12,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,945 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,946 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:12,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:12,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:12,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:12,947 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:12,947 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:12,947 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:12,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89, entries=150, sequenceid=54, filesize=11.6 K 2023-04-06 18:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:12,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/1660e73776434f2fba8b7595905c5a73 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73 2023-04-06 18:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:12,952 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:12,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:12,955 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,955 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:12,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ffe311f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:12,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:12,958 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,958 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:12,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:12,959 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:12,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:12,959 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73, entries=150, sequenceid=54, filesize=11.6 K 2023-04-06 18:58:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,961 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:12,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 50e5e6c3a16e85ee7fcdf95128e439dc in 906ms, sequenceid=54, compaction requested=true 2023-04-06 18:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:12,962 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:58:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,963 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:12,963 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:12,963 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:12,963 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,963 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:12,963 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=39.4 K 2023-04-06 18:58:12,963 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:12,964 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:12,964 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=34.8 K 2023-04-06 18:58:12,964 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4bf2d8d73e1c4709a345a5dbebe39db7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680807489776 2023-04-06 18:58:12,964 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7e9afbc384a348a0bfcec394bebd0738, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680807489776 2023-04-06 18:58:12,964 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a1bc0c152be6483abf8bea4a70275c4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=42, earliestPutTs=1680807489915 2023-04-06 18:58:12,964 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 03a1898d700c433ca8596ae468cb9fff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=42, earliestPutTs=1680807489915 2023-04-06 18:58:12,964 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 463c45516fde4fb6baccad6f7fccd584, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492045 2023-04-06 18:58:12,966 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0f59ddee6a7a4b58b53806b8adb0de89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492048 2023-04-06 18:58:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:12,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:12,966 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:12,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:12,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:12,966 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:12,966 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:12,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:12,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,972 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:12,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,977 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,979 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:12,979 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:12,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:12,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:12,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:12,984 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:12,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:12,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:12,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:12,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:12,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:12,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:12,984 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:12,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:12,984 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:13,005 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/292d4f5437de4019b974741b1a4b98f0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/292d4f5437de4019b974741b1a4b98f0 2023-04-06 18:58:13,009 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into 292d4f5437de4019b974741b1a4b98f0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:13,009 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:13,009 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=13, startTime=1680807492962; duration=0sec 2023-04-06 18:58:13,009 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:13,009 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:13,010 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:13,010 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:13,010 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:13,010 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=34.8 K 2023-04-06 18:58:13,010 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6d4f7a10a3a34ff69c0813cd9b2b0a57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680807489776 2023-04-06 18:58:13,011 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 72f70a2b30bd4202953225d2aa560b55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=42, earliestPutTs=1680807489915 2023-04-06 18:58:13,011 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1660e73776434f2fba8b7595905c5a73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492048 2023-04-06 18:58:13,018 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:13,043 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/d7a796fbc38c41cb94207c785f743fac as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d7a796fbc38c41cb94207c785f743fac 2023-04-06 18:58:13,049 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into d7a796fbc38c41cb94207c785f743fac(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:13,049 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:13,049 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=13, startTime=1680807492962; duration=0sec 2023-04-06 18:58:13,049 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:13,084 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:13,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:13,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:13,090 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:13,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:13,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:13,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:13,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:13,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:13,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:13,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:13,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:13,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:13,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:13,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:13,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:13,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:13,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:13,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:13,110 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:13,110 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:13,110 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:13,110 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:13,110 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:58:13,111 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:13,111 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:13,111 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:13,111 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:13,111 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:13,112 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:13,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:13,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:13,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807553285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807553286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807553287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807553293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807553293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,399 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/5a76aef8ce2846fe9b26a0c32581fbb7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/5a76aef8ce2846fe9b26a0c32581fbb7 2023-04-06 18:58:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807553394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807553394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807553394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,402 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into 5a76aef8ce2846fe9b26a0c32581fbb7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:13,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,402 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807553401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,402 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=13, startTime=1680807492962; duration=0sec 2023-04-06 18:58:13,402 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:13,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807553401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,530 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/9f8797fabf7041f7ad4c4412ad0ddbaf 2023-04-06 18:58:13,543 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/b997510b82be492898d79172243f6d5b 2023-04-06 18:58:13,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807553601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807553601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807553602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807553603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807553619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807553905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807553906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807553907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807553908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:13,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807553926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,960 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/4410e612b88b454ab6b2e6b2f6cad986 2023-04-06 18:58:13,963 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/9f8797fabf7041f7ad4c4412ad0ddbaf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf 2023-04-06 18:58:13,966 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf, entries=150, sequenceid=93, filesize=11.6 K 2023-04-06 18:58:13,967 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/b997510b82be492898d79172243f6d5b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b 2023-04-06 18:58:13,970 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b, entries=150, sequenceid=93, filesize=11.6 K 2023-04-06 18:58:13,971 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/4410e612b88b454ab6b2e6b2f6cad986 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986 2023-04-06 18:58:13,973 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986, entries=150, sequenceid=93, filesize=11.6 K 2023-04-06 18:58:13,974 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 50e5e6c3a16e85ee7fcdf95128e439dc in 864ms, sequenceid=93, compaction requested=false 2023-04-06 18:58:13,974 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:13,974 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:13,974 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:13,974 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:13,974 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,974 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:13,974 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:13,975 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:13,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:13,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:13,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3825286a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:13,977 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:13,977 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,977 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:13,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:13,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:13,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:13,978 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:13,978 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:13,978 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:13,978 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:13,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:13,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:13,980 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:13,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:13,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:13,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:13,981 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:13,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:13,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,982 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:13,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:13,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:13,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:13,984 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:13,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:13,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:13,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:13,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:13,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:13,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:13,985 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:13,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:13,985 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:13,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:13,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,085 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,096 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,097 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,097 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,097 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,097 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:14,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,121 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7af6e651ea244511a9b014e24f1755fd 2023-04-06 18:58:14,166 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/a4700d78cab443b99556e84b8623fca0 2023-04-06 18:58:14,184 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/d5fc468f940c4b21bade040df9973ef6 2023-04-06 18:58:14,191 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7af6e651ea244511a9b014e24f1755fd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd 2023-04-06 18:58:14,196 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd, entries=150, sequenceid=105, filesize=11.6 K 2023-04-06 18:58:14,196 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/a4700d78cab443b99556e84b8623fca0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0 2023-04-06 18:58:14,201 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0, entries=150, sequenceid=105, filesize=11.6 K 2023-04-06 18:58:14,202 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/d5fc468f940c4b21bade040df9973ef6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6 2023-04-06 18:58:14,207 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6, entries=150, sequenceid=105, filesize=11.6 K 2023-04-06 18:58:14,207 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 50e5e6c3a16e85ee7fcdf95128e439dc in 110ms, sequenceid=105, compaction requested=true 2023-04-06 18:58:14,207 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:14,208 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,208 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,211 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,212 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,212 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302688c3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,215 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,215 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:14,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,231 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,231 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,231 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,246 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,247 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,248 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:14,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,255 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,256 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,256 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,356 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,359 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,361 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,361 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,361 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:14,361 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,362 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52bc858b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,365 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:14,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,366 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,366 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,366 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,366 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,366 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,369 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,370 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,370 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,370 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,371 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:14,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:14,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:14,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:14,470 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,486 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,486 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,489 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,490 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680807494483, max timestamp=1680807494505Num uniques 859; 2023-04-06 18:58:14,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 18:58:14,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680807494483, max timestamp=1680807494505Num uniques 859; 2023-04-06 18:58:14,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 18:58:14,509 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,509 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,510 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:14,510 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807494497, max timestamp=1680807494505Num uniques 859; 2023-04-06 18:58:14,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:14,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,511 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,512 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,512 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc4200a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,515 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:14,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,517 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,517 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,518 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,522 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,522 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,523 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,522 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,523 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,523 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:14,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807554542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807554558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807554558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807554558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807554559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,623 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,627 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,629 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,629 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,629 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,630 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,634 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,634 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,634 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:14,634 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,639 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,639 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d332580[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,648 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,648 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,650 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,651 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,651 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,651 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,656 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,660 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:14,661 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,661 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807554659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807554668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807554670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807554677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807554679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,761 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,770 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,771 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,771 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,772 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,774 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,774 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,774 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,774 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:14,775 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,776 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,776 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d7dc75[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,778 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,778 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,778 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,779 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:14,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,780 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,780 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,780 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,780 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,780 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,781 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,781 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,781 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,782 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,783 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,784 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,784 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,785 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,785 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,785 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,790 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807554866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,885 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:14,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:14,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:14,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807554879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:14,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:14,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/ab1b890bfec84a1d90920e12a0dbe65b 2023-04-06 18:58:14,898 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:14,898 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,899 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:14,899 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:14,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807554888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,901 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,902 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,902 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,903 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,903 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:14,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807554893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:14,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807554893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:14,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:14,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:14,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:14,916 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:14,919 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:14,919 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:14,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,922 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:14,923 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:14,923 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:14,924 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:14,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f718819[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:14,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:14,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:14,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:14,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:14,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,930 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:14,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:14,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,931 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:14,931 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:14,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:14,937 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:14,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:14,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:14,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:14,937 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:14,947 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,947 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,947 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:14,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:14,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:14,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:14,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:14,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:14,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:14,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:14,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,951 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:14,951 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:14,951 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:14,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:14,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:14,952 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:14,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:14,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,052 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,056 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,057 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,059 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,068 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,073 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,073 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248238d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,078 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,079 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,079 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,080 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,080 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,080 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,082 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,082 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,086 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,086 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,091 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,094 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,106 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,106 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,106 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,106 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,106 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,107 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807555181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807555191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,207 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807555201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807555206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807555207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,211 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,231 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,235 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,236 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56831961[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,237 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,237 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,238 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,238 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,238 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,240 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,241 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,241 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,242 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,243 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,244 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,245 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,245 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,345 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,348 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,349 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,350 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,350 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,351 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,351 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c043b0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,364 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,370 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,370 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,370 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,373 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,373 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,373 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,374 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,376 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:15,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,377 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,377 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/778df82fb77041778e30167693b7dbf9 2023-04-06 18:58:15,477 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,480 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,482 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,482 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,484 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,484 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,485 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,485 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,486 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,486 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@528226d1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,487 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,487 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,488 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,488 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,488 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,488 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,490 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,490 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,490 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,491 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,494 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,494 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,495 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,494 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,495 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,495 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,546 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-06 18:58:15,555 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584] to archive 2023-04-06 18:58:15,556 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:15,557 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4bf2d8d73e1c4709a345a5dbebe39db7 2023-04-06 18:58:15,558 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a1bc0c152be6483abf8bea4a70275c4e 2023-04-06 18:58:15,558 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/463c45516fde4fb6baccad6f7fccd584 2023-04-06 18:58:15,565 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89] to archive 2023-04-06 18:58:15,565 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:15,567 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/7e9afbc384a348a0bfcec394bebd0738 2023-04-06 18:58:15,568 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/03a1898d700c433ca8596ae468cb9fff 2023-04-06 18:58:15,569 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/0f59ddee6a7a4b58b53806b8adb0de89 2023-04-06 18:58:15,575 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73] to archive 2023-04-06 18:58:15,575 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:15,577 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6d4f7a10a3a34ff69c0813cd9b2b0a57 2023-04-06 18:58:15,578 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/72f70a2b30bd4202953225d2aa560b55 2023-04-06 18:58:15,578 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1660e73776434f2fba8b7595905c5a73 2023-04-06 18:58:15,595 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,597 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,599 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,599 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,599 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,599 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,599 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,600 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,600 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11952347[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,602 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,602 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,602 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,602 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,604 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,604 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,604 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,605 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,605 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,607 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,608 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,608 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,608 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,608 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807555697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807555698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,708 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,711 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,712 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,712 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,713 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,713 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,713 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,713 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:15,713 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:15,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,714 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,715 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,715 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:15,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:15,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f136805[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:15,716 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,716 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:15,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:15,716 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:15,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:15,717 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,717 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:15,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:15,719 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:15,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:15,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:15,719 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:15,719 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:15,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807555713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807555714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807555716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,720 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:15,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:15,721 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,721 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:15,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:15,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:15,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:15,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:15,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:15,724 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:15,724 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:15,803 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/84b70b8ef37247a4bbecf652ecd0fa1a 2023-04-06 18:58:15,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/ab1b890bfec84a1d90920e12a0dbe65b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b 2023-04-06 18:58:15,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b, entries=200, sequenceid=117, filesize=13.9 K 2023-04-06 18:58:15,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/778df82fb77041778e30167693b7dbf9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9 2023-04-06 18:58:15,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9, entries=150, sequenceid=117, filesize=11.6 K 2023-04-06 18:58:15,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/84b70b8ef37247a4bbecf652ecd0fa1a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a 2023-04-06 18:58:15,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a, entries=150, sequenceid=117, filesize=11.6 K 2023-04-06 18:58:15,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=138.65 KB/141980 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1398ms, sequenceid=117, compaction requested=true 2023-04-06 18:58:15,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:15,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:15,822 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:15,822 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:15,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:15,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:15,823 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49987 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:15,823 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:15,823 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:15,823 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:15,823 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,823 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,823 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/5a76aef8ce2846fe9b26a0c32581fbb7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=48.8 K 2023-04-06 18:58:15,823 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/292d4f5437de4019b974741b1a4b98f0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=46.5 K 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 5a76aef8ce2846fe9b26a0c32581fbb7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492048 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 292d4f5437de4019b974741b1a4b98f0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492048 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9f8797fabf7041f7ad4c4412ad0ddbaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680807492126 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b997510b82be492898d79172243f6d5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680807492126 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7af6e651ea244511a9b014e24f1755fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807493286 2023-04-06 18:58:15,824 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:15,824 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a4700d78cab443b99556e84b8623fca0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807493286 2023-04-06 18:58:15,825 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ab1b890bfec84a1d90920e12a0dbe65b, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494412 2023-04-06 18:58:15,825 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 778df82fb77041778e30167693b7dbf9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494423 2023-04-06 18:58:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:15,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:15,827 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:15,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:15,827 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:15,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,829 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:15,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:15,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:15,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:15,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:15,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:15,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:15,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:15,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:15,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:15,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:15,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:15,831 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:15,831 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=138.65 KB heapSize=385.77 KB 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:15,831 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:15,832 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:15,833 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#406 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:15,833 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#407 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:15,868 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/ca8062e8a1824f31a007ba7ff31c0af5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ca8062e8a1824f31a007ba7ff31c0af5 2023-04-06 18:58:15,874 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d0d2fbba26d7429f8f77e94477eec13b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d0d2fbba26d7429f8f77e94477eec13b 2023-04-06 18:58:15,875 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into ca8062e8a1824f31a007ba7ff31c0af5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:15,875 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:15,875 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=12, startTime=1680807495822; duration=0sec 2023-04-06 18:58:15,875 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:15,875 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:15,877 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:15,877 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:15,877 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:15,877 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d7a796fbc38c41cb94207c785f743fac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=46.5 K 2023-04-06 18:58:15,877 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d7a796fbc38c41cb94207c785f743fac, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1680807492048 2023-04-06 18:58:15,877 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4410e612b88b454ab6b2e6b2f6cad986, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680807492126 2023-04-06 18:58:15,878 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into d0d2fbba26d7429f8f77e94477eec13b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:15,878 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:15,878 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=12, startTime=1680807495822; duration=0sec 2023-04-06 18:58:15,878 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:15,879 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d5fc468f940c4b21bade040df9973ef6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1680807493286 2023-04-06 18:58:15,881 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 84b70b8ef37247a4bbecf652ecd0fa1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494423 2023-04-06 18:58:15,885 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:15,892 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/9d32fdc7186b4d46955fbb2c20bf122e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9d32fdc7186b4d46955fbb2c20bf122e 2023-04-06 18:58:15,895 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into 9d32fdc7186b4d46955fbb2c20bf122e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:15,895 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:15,895 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=12, startTime=1680807495822; duration=0sec 2023-04-06 18:58:15,895 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:16,265 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/1163ab436240451e8dd03f4b1a766d1f 2023-04-06 18:58:16,673 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/8d06b28ca09245a3becae4419777648b 2023-04-06 18:58:16,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:16,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:16,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807556718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807556722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807556723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807556726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807556727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807556823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:16,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:16,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807556827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807557026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807557032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,085 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/34bcd065460f4677ab26d1d8f08f9330 2023-04-06 18:58:17,089 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/1163ab436240451e8dd03f4b1a766d1f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f 2023-04-06 18:58:17,092 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f, entries=150, sequenceid=152, filesize=11.8 K 2023-04-06 18:58:17,093 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/8d06b28ca09245a3becae4419777648b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b 2023-04-06 18:58:17,097 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b, entries=150, sequenceid=152, filesize=11.8 K 2023-04-06 18:58:17,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/34bcd065460f4677ab26d1d8f08f9330 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330 2023-04-06 18:58:17,102 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330, entries=150, sequenceid=152, filesize=11.8 K 2023-04-06 18:58:17,102 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~385.72 KB/394976, currentSize=46.96 KB/48090 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1271ms, sequenceid=152, compaction requested=false 2023-04-06 18:58:17,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:17,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:17,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:17,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:17,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:17,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:17,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:17,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:17,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:17,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c6d7d94[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:17,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:17,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:17,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:17,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:17,106 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:17,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:17,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:17,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:17,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:17,108 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:17,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:17,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:17,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:17,108 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:17,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,109 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:17,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:17,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:17,111 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:17,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:17,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:17,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:17,115 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:17,115 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:17,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:17,215 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:17,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:17,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:17,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:17,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:17,218 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:17,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:17,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:17,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:17,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:17,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:17,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:17,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:17,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:17,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:17,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:17,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:17,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:17,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:17,236 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:17,236 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:17,236 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:17,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:17,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:17,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:17,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:17,248 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:17,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:17,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:17,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:17,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:17,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:17,249 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:17,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:17,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:17,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807497448, max timestamp=1680807497471Num uniques 859; 2023-04-06 18:58:17,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:17,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807497448, max timestamp=1680807497471Num uniques 859; 2023-04-06 18:58:17,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:17,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807497448, max timestamp=1680807497471Num uniques 859; 2023-04-06 18:58:17,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:17,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807557575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807557576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807557685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807557686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,698 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/e631cfa391654626946cb22d0cccb005 2023-04-06 18:58:17,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807557895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:17,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:17,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807557900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,134 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/c03480e63575499c8efe8d82d7e16c48 2023-04-06 18:58:18,183 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/7381ec0fea3e4526bb3dfe5ea069f264 2023-04-06 18:58:18,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/e631cfa391654626946cb22d0cccb005 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005 2023-04-06 18:58:18,193 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005, entries=150, sequenceid=165, filesize=11.8 K 2023-04-06 18:58:18,195 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/c03480e63575499c8efe8d82d7e16c48 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48 2023-04-06 18:58:18,202 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48, entries=150, sequenceid=165, filesize=11.8 K 2023-04-06 18:58:18,205 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/7381ec0fea3e4526bb3dfe5ea069f264 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264 2023-04-06 18:58:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807558204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,223 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264, entries=150, sequenceid=165, filesize=11.8 K 2023-04-06 18:58:18,224 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 50e5e6c3a16e85ee7fcdf95128e439dc in 976ms, sequenceid=165, compaction requested=true 2023-04-06 18:58:18,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:18,224 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,224 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,224 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,224 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:18,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:58:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:18,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1a7866[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,234 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,234 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,235 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,235 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,237 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,237 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:18,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,238 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:18,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:18,242 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,242 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:18,242 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:18,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807558302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,343 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:18,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:18,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:18,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:18,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:18,347 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:18,347 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,347 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:18,347 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:18,347 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,347 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,348 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,348 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,349 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:18,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:18,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:18,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:18,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:18,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,363 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,363 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adebd9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,367 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,367 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,373 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,373 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,375 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,380 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,380 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,386 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,386 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:18,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:18,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:18,400 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:18,400 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:18,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807558412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,500 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:18,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:18,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:18,518 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:18,518 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,518 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:18,518 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:18,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,519 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,520 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,520 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:18,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:18,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:18,521 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:18,521 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,521 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:18,522 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:18,523 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,523 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:18,523 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:18,523 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,523 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,525 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,525 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,525 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,525 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@433af869[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,526 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,528 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,528 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,529 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,530 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,530 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,530 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,530 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:18,531 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,531 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,531 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,531 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:18,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:18,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,534 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,535 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:18,535 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807558618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,635 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:18,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:18,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:18,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:18,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:18,638 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:18,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:18,638 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:18,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,642 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:18,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:18,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:18,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:18,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:18,644 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,644 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,645 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,645 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542cbc47[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,649 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,652 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,652 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:18,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,656 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:18,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:18,658 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:18,658 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:18,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/4fc708b0f66545b7aa625aba039ebb1b 2023-04-06 18:58:18,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/3d5c420cf6fa496685fe30e25b103d66 2023-04-06 18:58:18,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/0c9ea399e4a549b595ad345748a6febd 2023-04-06 18:58:18,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807558716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/4fc708b0f66545b7aa625aba039ebb1b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b 2023-04-06 18:58:18,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b, entries=200, sequenceid=203, filesize=14.1 K 2023-04-06 18:58:18,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/3d5c420cf6fa496685fe30e25b103d66 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66 2023-04-06 18:58:18,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66, entries=150, sequenceid=203, filesize=11.8 K 2023-04-06 18:58:18,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807558734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,737 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/0c9ea399e4a549b595ad345748a6febd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd 2023-04-06 18:58:18,742 DEBUG [Thread-3117] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:58:18,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd, entries=150, sequenceid=203, filesize=11.8 K 2023-04-06 18:58:18,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 50e5e6c3a16e85ee7fcdf95128e439dc in 520ms, sequenceid=203, compaction requested=true 2023-04-06 18:58:18,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:18,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:18,746 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:18,746 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:18,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:18,746 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:18,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50522 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:18,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:18,747 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,747 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ca8062e8a1824f31a007ba7ff31c0af5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=49.3 K 2023-04-06 18:58:18,747 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48132 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:18,747 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:18,747 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,748 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d0d2fbba26d7429f8f77e94477eec13b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=47.0 K 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ca8062e8a1824f31a007ba7ff31c0af5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494423 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d0d2fbba26d7429f8f77e94477eec13b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494423 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 1163ab436240451e8dd03f4b1a766d1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680807494554 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8d06b28ca09245a3becae4419777648b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680807494554 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e631cfa391654626946cb22d0cccb005, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680807496716 2023-04-06 18:58:18,748 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c03480e63575499c8efe8d82d7e16c48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680807496716 2023-04-06 18:58:18,749 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4fc708b0f66545b7aa625aba039ebb1b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:18,749 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3d5c420cf6fa496685fe30e25b103d66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:18,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:18,753 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:18,754 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:18,758 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:18,759 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#418 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:18,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:18,768 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#419 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:18,802 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:18,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:18,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:18,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,804 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:18,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:18,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:18,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:18,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,807 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:18,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:18,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:18,810 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:18,811 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:18,811 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,812 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,812 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@381927ec[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,814 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,814 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,814 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,815 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,817 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,817 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:18,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,821 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,823 WARN [DataStreamer for file /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d684631bc138486b8a52db5da65a5d53] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-06 18:58:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,830 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d684631bc138486b8a52db5da65a5d53 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d684631bc138486b8a52db5da65a5d53 2023-04-06 18:58:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807498809, max timestamp=1680807498834Num uniques 859; 2023-04-06 18:58:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807498809, max timestamp=1680807498834Num uniques 859; 2023-04-06 18:58:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:18,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:18,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807498809, max timestamp=1680807498834Num uniques 859; 2023-04-06 18:58:18,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:18,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:18,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:18,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,839 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:18,839 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:18,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:18,842 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into d684631bc138486b8a52db5da65a5d53(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:18,842 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:18,842 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=12, startTime=1680807498746; duration=0sec 2023-04-06 18:58:18,842 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:18,842 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:18,843 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48132 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:18,844 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:18,844 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,844 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9d32fdc7186b4d46955fbb2c20bf122e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=47.0 K 2023-04-06 18:58:18,844 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9d32fdc7186b4d46955fbb2c20bf122e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680807494423 2023-04-06 18:58:18,844 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 34bcd065460f4677ab26d1d8f08f9330, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680807494554 2023-04-06 18:58:18,845 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7381ec0fea3e4526bb3dfe5ea069f264, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680807496716 2023-04-06 18:58:18,845 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0c9ea399e4a549b595ad345748a6febd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:18,870 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:18,887 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/cc034a1c07484d82816ede13c65dc862 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/cc034a1c07484d82816ede13c65dc862 2023-04-06 18:58:18,890 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into cc034a1c07484d82816ede13c65dc862(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:18,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:18,890 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=12, startTime=1680807498746; duration=0sec 2023-04-06 18:58:18,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:18,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807558926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807558927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:18,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807558929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,939 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:18,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:18,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:18,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:18,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:18,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:18,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:18,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:18,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:18,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:18,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,960 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:18,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:18,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:18,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:18,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:18,966 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:18,972 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,972 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:18,972 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:18,972 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,972 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:18,973 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:18,980 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:18,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5082e015[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:18,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:18,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:18,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:18,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:18,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:18,989 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,989 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:18,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:18,997 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:18,997 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:18,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:18,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:18,999 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:18,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:18,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:18,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:18,999 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,006 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,013 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,013 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,014 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,014 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,014 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807559038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807559038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,114 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,118 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,118 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,119 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,120 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,120 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,122 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,131 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,131 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446fb3dc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,133 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,134 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,134 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,135 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,135 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,135 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,135 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,137 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,137 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,137 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,138 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,139 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,139 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,140 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,142 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,143 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,143 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,212 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/a7a6e727560e4508b95ce91bd9132d01 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a7a6e727560e4508b95ce91bd9132d01 2023-04-06 18:58:19,215 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into a7a6e727560e4508b95ce91bd9132d01(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:19,215 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:19,215 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=12, startTime=1680807498746; duration=0sec 2023-04-06 18:58:19,215 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:19,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/c177cd52943a49bfac0e5b3447ca3eba 2023-04-06 18:58:19,243 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,245 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,246 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,246 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,248 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807559245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,248 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807559245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,249 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,250 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7590e192[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,250 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,251 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,251 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,251 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,252 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,253 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,253 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,253 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,255 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,255 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,255 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,256 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,258 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,259 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,260 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,260 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,360 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,363 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,363 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,365 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,365 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,365 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,365 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,365 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,366 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,366 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c79e8b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,368 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,369 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,369 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,383 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,383 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,386 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,386 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,392 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,393 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,393 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807559436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,493 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,496 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,496 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,497 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,497 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,498 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,498 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,499 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,499 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4382444e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,505 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,505 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,506 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,507 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,508 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,514 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,514 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807559551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807559551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,614 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,617 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,617 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,622 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/6d38537e3f9c435f89bc31aef15e84f1 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7acaeeb1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,625 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,626 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,626 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,628 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,628 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,629 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,630 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,632 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,634 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/120b19399c774e1097ef13ee3f5821cc 2023-04-06 18:58:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,636 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,636 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,637 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/c177cd52943a49bfac0e5b3447ca3eba as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba 2023-04-06 18:58:19,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba, entries=200, sequenceid=214, filesize=14.1 K 2023-04-06 18:58:19,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/6d38537e3f9c435f89bc31aef15e84f1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1 2023-04-06 18:58:19,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1, entries=150, sequenceid=214, filesize=11.8 K 2023-04-06 18:58:19,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/120b19399c774e1097ef13ee3f5821cc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc 2023-04-06 18:58:19,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc, entries=150, sequenceid=214, filesize=11.8 K 2023-04-06 18:58:19,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 50e5e6c3a16e85ee7fcdf95128e439dc in 897ms, sequenceid=214, compaction requested=false 2023-04-06 18:58:19,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:19,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:19,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:19,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:19,737 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,742 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,744 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,744 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,745 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,745 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,745 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,745 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,745 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,746 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,746 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,747 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8241c0b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,747 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,749 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,750 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,750 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,752 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,752 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,752 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,752 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,752 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,753 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,755 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,756 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,756 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807559773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,856 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,859 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,862 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,862 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,866 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,866 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,866 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,866 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,866 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,868 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30aff842[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,872 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,872 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,873 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,873 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,873 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,875 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,875 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,878 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,879 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:19,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807559879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,883 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,883 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,983 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:19,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:19,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:19,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:19,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:19,986 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:19,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:19,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:19,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:19,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:19,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:19,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:19,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:19,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:19,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:19,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:19,988 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,989 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:19,989 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:19,989 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:19,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:19,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:19,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:19,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:19,990 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,990 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:19,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:19,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615e6a2d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:19,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,991 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:19,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:19,992 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:19,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:19,994 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:19,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:19,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:19,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:19,994 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:19,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:19,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:19,996 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:19,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:19,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:19,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:19,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:19,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:19,999 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:19,999 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807560058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807560058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807560082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,099 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,102 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,104 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,105 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,106 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4cbfd1[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,107 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,110 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,110 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,111 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,112 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,114 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,114 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/a62d03a26253438a84c48d46563a4fae 2023-04-06 18:58:20,214 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,217 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,219 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,219 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,219 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,219 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64202617[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,223 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,225 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,225 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,227 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,229 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,229 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,329 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,332 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,334 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,336 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,336 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b02af39[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,339 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,340 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,341 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,341 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,342 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,344 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,345 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,345 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57130 deadline: 1680807560387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,445 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,447 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807560443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,451 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,451 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,452 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,452 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,452 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,452 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,452 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,452 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,452 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399fa776[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,456 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,457 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,457 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,459 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,460 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,460 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,461 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,463 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,464 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,465 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,465 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,465 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,465 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/550fb710de4f47dbbac04f1dbb88be03 2023-04-06 18:58:20,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/29888e3074c3441e823bfbc0bf7e7f98 2023-04-06 18:58:20,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/a62d03a26253438a84c48d46563a4fae as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae 2023-04-06 18:58:20,565 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae, entries=200, sequenceid=253, filesize=14.1 K 2023-04-06 18:58:20,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/550fb710de4f47dbbac04f1dbb88be03 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03 2023-04-06 18:58:20,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,568 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,568 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/29888e3074c3441e823bfbc0bf7e7f98 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,571 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,571 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,571 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,571 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,571 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108435a3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 18:58:20,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 50e5e6c3a16e85ee7fcdf95128e439dc in 841ms, sequenceid=253, compaction requested=true 2023-04-06 18:58:20,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:20,576 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:20,576 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,577 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,577 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41162 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:20,577 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,577 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,578 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a7a6e727560e4508b95ce91bd9132d01, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=40.2 K 2023-04-06 18:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a7a6e727560e4508b95ce91bd9132d01, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:20,578 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,578 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d684631bc138486b8a52db5da65a5d53, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=35.5 K 2023-04-06 18:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting c177cd52943a49bfac0e5b3447ca3eba, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680807498284 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d684631bc138486b8a52db5da65a5d53, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,578 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a62d03a26253438a84c48d46563a4fae, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:20,579 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6d38537e3f9c435f89bc31aef15e84f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680807498284 2023-04-06 18:58:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,579 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 550fb710de4f47dbbac04f1dbb88be03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,580 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,580 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,580 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,580 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,580 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,582 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,582 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,582 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,583 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,593 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#427 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:20,595 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,599 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,601 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,601 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,602 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,602 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,602 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,637 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/98c04e4b54de466c9b5d3dd0782dec97 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/98c04e4b54de466c9b5d3dd0782dec97 2023-04-06 18:58:20,644 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into 98c04e4b54de466c9b5d3dd0782dec97(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:20,644 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,644 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=13, startTime=1680807500576; duration=0sec 2023-04-06 18:58:20,644 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:20,644 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:20,646 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:20,646 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:20,646 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,646 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/cc034a1c07484d82816ede13c65dc862, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=35.5 K 2023-04-06 18:58:20,647 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting cc034a1c07484d82816ede13c65dc862, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680807497532 2023-04-06 18:58:20,647 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 120b19399c774e1097ef13ee3f5821cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680807498284 2023-04-06 18:58:20,647 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 29888e3074c3441e823bfbc0bf7e7f98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:20,648 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/f7838526dfc5486da707428dc1656c25 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/f7838526dfc5486da707428dc1656c25 2023-04-06 18:58:20,663 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into f7838526dfc5486da707428dc1656c25(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:20,664 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,664 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=13, startTime=1680807500576; duration=0sec 2023-04-06 18:58:20,664 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:20,665 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:20,702 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,704 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,705 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,705 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,705 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,705 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,706 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,707 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:20,707 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,713 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/0c1b7bcdae6149aca2ed53a54bc4442c 2023-04-06 18:58:20,723 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/5e9940aed46042d18bafea21762107af 2023-04-06 18:58:20,729 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/35ca54fd363e4b41add740ef22073589 2023-04-06 18:58:20,733 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/0c1b7bcdae6149aca2ed53a54bc4442c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c 2023-04-06 18:58:20,735 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c, entries=150, sequenceid=265, filesize=11.9 K 2023-04-06 18:58:20,736 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/5e9940aed46042d18bafea21762107af as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af 2023-04-06 18:58:20,738 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af, entries=150, sequenceid=265, filesize=11.9 K 2023-04-06 18:58:20,739 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/35ca54fd363e4b41add740ef22073589 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589 2023-04-06 18:58:20,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589, entries=150, sequenceid=265, filesize=11.9 K 2023-04-06 18:58:20,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 50e5e6c3a16e85ee7fcdf95128e439dc in 35ms, sequenceid=265, compaction requested=false 2023-04-06 18:58:20,742 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,743 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,743 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,744 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,744 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,744 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,744 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75f1b93a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,745 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,745 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,746 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,746 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,747 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,747 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,748 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,751 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,751 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,751 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,752 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,755 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:20,757 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,757 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,857 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,859 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,859 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,860 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:20,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,861 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,861 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,862 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,862 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,862 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,864 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb4abda[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,864 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,864 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,864 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,865 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,865 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,865 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,865 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,865 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,868 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,868 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,868 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,868 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,868 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,869 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,869 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,869 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,872 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,873 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,873 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,873 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,874 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:20,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:20,915 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:20,915 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,915 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:20,915 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502 2023-04-06 18:58:20,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/b03ceaab913d44cfb09de99163a5896b 2023-04-06 18:58:20,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/ed5c04e7fd054bdabf7ff937479e10f7 2023-04-06 18:58:20,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502 2023-04-06 18:58:20,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502, entries=200, sequenceid=276, filesize=14.3 K 2023-04-06 18:58:20,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/b03ceaab913d44cfb09de99163a5896b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b 2023-04-06 18:58:20,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b, entries=150, sequenceid=276, filesize=11.9 K 2023-04-06 18:58:20,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/ed5c04e7fd054bdabf7ff937479e10f7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7 2023-04-06 18:58:20,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7, entries=150, sequenceid=276, filesize=11.9 K 2023-04-06 18:58:20,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=33.54 KB/34350 for 50e5e6c3a16e85ee7fcdf95128e439dc in 35ms, sequenceid=276, compaction requested=true 2023-04-06 18:58:20,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:20,959 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:20,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:20,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:58:20,960 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:20,960 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:20,960 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,960 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/f7838526dfc5486da707428dc1656c25, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=38.3 K 2023-04-06 18:58:20,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:20,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:20,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:20,965 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,965 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:20,965 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,965 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:20,965 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:20,966 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f7838526dfc5486da707428dc1656c25, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:20,967 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0c1b7bcdae6149aca2ed53a54bc4442c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680807499758 2023-04-06 18:58:20,967 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b5fb8a3c5e0d4d7ca0ef908bd92f1502, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680807500895 2023-04-06 18:58:20,974 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:20,975 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:20,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:20,980 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:20,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:20,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:20,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:20,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:20,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:20,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:20,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:20,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:20,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:20,983 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,984 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:20,984 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:20,984 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:20,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:20,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:20,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:20,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:20,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:20,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5411cd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:20,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:20,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:20,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,987 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2fd2654a60f24e4b880aaa51bf6dd709 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2fd2654a60f24e4b880aaa51bf6dd709 2023-04-06 18:58:20,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:20,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:20,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:20,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:20,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,992 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into 2fd2654a60f24e4b880aaa51bf6dd709(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:20,992 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:20,992 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=13, startTime=1680807500951; duration=0sec 2023-04-06 18:58:20,992 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 18:58:20,992 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-06 18:58:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:20,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:20,992 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:20,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:20,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:20,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:20,993 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:20,993 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:58:20,993 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:58:20,993 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. because compaction request was cancelled 2023-04-06 18:58:20,993 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:20,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,995 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:20,996 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:20,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:20,997 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:20,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:20,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:20,997 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:20,997 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:20,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:20,998 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:20,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,002 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:21,002 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:21,002 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,003 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/98c04e4b54de466c9b5d3dd0782dec97, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=35.9 K 2023-04-06 18:58:21,003 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 98c04e4b54de466c9b5d3dd0782dec97, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:21,003 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 5e9940aed46042d18bafea21762107af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680807499758 2023-04-06 18:58:21,005 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting b03ceaab913d44cfb09de99163a5896b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680807500899 2023-04-06 18:58:21,013 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:21,026 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/b099ad733bf44f4ca0885b39454a1243 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b099ad733bf44f4ca0885b39454a1243 2023-04-06 18:58:21,029 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into b099ad733bf44f4ca0885b39454a1243(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:21,029 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:21,029 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=13, startTime=1680807500959; duration=0sec 2023-04-06 18:58:21,029 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:21,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807561070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807500993, max timestamp=1680807501075Num uniques 859; 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807500993, max timestamp=1680807501075Num uniques 859; 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807500993, max timestamp=1680807501075Num uniques 859; 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:21,090 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/65dc7952796d41918e54a0a49c35f4b1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/65dc7952796d41918e54a0a49c35f4b1 2023-04-06 18:58:21,093 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into 65dc7952796d41918e54a0a49c35f4b1(size=12.1 K), total size for store is 35.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:21,093 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:21,093 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=13, startTime=1680807500577; duration=0sec 2023-04-06 18:58:21,093 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:21,097 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,100 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,102 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4c14a9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,114 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,114 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,114 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,115 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,115 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,116 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,116 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,116 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,116 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,116 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,117 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,121 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,122 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,123 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,123 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,124 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,124 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,124 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807561140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807561142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807561177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,224 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,227 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,227 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,228 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,230 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,230 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,230 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,230 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,230 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,230 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,231 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,231 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,232 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468c4233[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,232 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,233 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,233 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,233 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,234 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,234 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,235 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,236 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,236 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,236 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,236 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,236 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,237 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,239 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,239 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,239 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807561248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807561248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,339 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,342 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,343 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,343 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,344 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,344 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,345 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,345 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,345 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,345 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,345 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,345 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,346 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2888cb43[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,356 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,356 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,356 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,356 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,358 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,358 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,358 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,361 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,361 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,362 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,362 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,362 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2434ea40d3b2460abf25fc556b5ba75b 2023-04-06 18:58:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807561383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/4c1bc48e5ada4e04b5e59ec37a78bc18 2023-04-06 18:58:21,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/9487e6112d4a43eb89e33b727a3b6600 2023-04-06 18:58:21,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2434ea40d3b2460abf25fc556b5ba75b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b 2023-04-06 18:58:21,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b, entries=200, sequenceid=286, filesize=14.3 K 2023-04-06 18:58:21,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/4c1bc48e5ada4e04b5e59ec37a78bc18 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18 2023-04-06 18:58:21,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18, entries=150, sequenceid=286, filesize=11.9 K 2023-04-06 18:58:21,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/9487e6112d4a43eb89e33b727a3b6600 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600 2023-04-06 18:58:21,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600, entries=150, sequenceid=286, filesize=11.9 K 2023-04-06 18:58:21,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 50e5e6c3a16e85ee7fcdf95128e439dc in 460ms, sequenceid=286, compaction requested=true 2023-04-06 18:58:21,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:21,424 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 18:58:21,424 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 18:58:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:21,424 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. because compaction request was cancelled 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:58:21,425 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. because compaction request was cancelled 2023-04-06 18:58:21,426 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:21,426 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:21,426 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,426 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/65dc7952796d41918e54a0a49c35f4b1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=47.8 K 2023-04-06 18:58:21,426 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 65dc7952796d41918e54a0a49c35f4b1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680807498823 2023-04-06 18:58:21,426 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 35ca54fd363e4b41add740ef22073589, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680807499758 2023-04-06 18:58:21,427 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ed5c04e7fd054bdabf7ff937479e10f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680807500899 2023-04-06 18:58:21,427 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9487e6112d4a43eb89e33b727a3b6600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680807500929 2023-04-06 18:58:21,432 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:21,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:21,462 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-06 18:58:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807501119, max timestamp=1680807501462Num uniques 1068; 2023-04-06 18:58:21,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:58:21,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807501119, max timestamp=1680807501462Num uniques 1068; 2023-04-06 18:58:21,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:58:21,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680807501119, max timestamp=1680807501462Num uniques 1068; 2023-04-06 18:58:21,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:21,465 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:21,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,468 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,468 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,469 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,469 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,470 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,470 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,470 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,472 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,472 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,473 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd6de6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,475 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,476 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,476 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,478 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,478 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,478 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,479 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,481 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,481 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,481 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,482 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807501500, max timestamp=1680807501513Num uniques 859; 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807501500, max timestamp=1680807501513Num uniques 859; 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807501500, max timestamp=1680807501513Num uniques 859; 2023-04-06 18:58:21,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:21,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57206 deadline: 1680807561560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807501550, max timestamp=1680807501566Num uniques 1118; 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807501550, max timestamp=1680807501566Num uniques 1118; 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807501550, max timestamp=1680807501566Num uniques 1118; 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 18:58:21,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 18:58:21,581 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,584 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,585 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,585 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,586 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,587 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,587 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594ac8ec[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,589 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,589 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,590 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,592 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,592 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,592 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,592 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,592 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,593 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,593 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,593 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,593 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,595 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,596 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,597 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,597 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,596 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,597 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,597 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,597 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807501622, max timestamp=1680807501666Num uniques 1027; 2023-04-06 18:58:21,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:21,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807501622, max timestamp=1680807501666Num uniques 1027; 2023-04-06 18:58:21,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:21,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807501622, max timestamp=1680807501666Num uniques 1027; 2023-04-06 18:58:21,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:21,697 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,699 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,699 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,700 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,700 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,700 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,700 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,701 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,701 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,701 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,701 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,701 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,701 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,702 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,702 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,702 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,702 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d73942c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,704 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,704 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,705 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,708 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,708 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,708 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,709 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,709 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,710 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,711 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,711 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,711 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,711 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,712 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8500, totalHeapSize=19584, min timestamp=1680807501696, max timestamp=1680807501724Num uniques 1036; 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47430, new segment data size=8500, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8499, totalHeapSize=19584, min timestamp=1680807501696, max timestamp=1680807501724Num uniques 1036; 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47429, new segment data size=8499, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8501, totalHeapSize=19584, min timestamp=1680807501696, max timestamp=1680807501724Num uniques 1036; 2023-04-06 18:58:21,733 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47431, new segment data size=8501, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 18:58:21,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807501762, max timestamp=1680807501776Num uniques 1045; 2023-04-06 18:58:21,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807501762, max timestamp=1680807501776Num uniques 1045; 2023-04-06 18:58:21,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680807501762, max timestamp=1680807501776Num uniques 1045; 2023-04-06 18:58:21,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:58:21,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:58:21,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-06 18:58:21,812 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,814 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,816 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,817 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,817 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,818 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,818 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13cf9a12[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,819 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,819 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,819 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,820 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,821 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,822 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,822 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,822 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,824 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,825 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,825 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,825 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,825 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57182 deadline: 1680807561834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680807501802, max timestamp=1680807501834Num uniques 1154; 2023-04-06 18:58:21,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-06 18:58:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680807501802, max timestamp=1680807501834Num uniques 1154; 2023-04-06 18:58:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1680807501802, max timestamp=1680807501834Num uniques 1154; 2023-04-06 18:58:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-06 18:58:21,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=11624, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-06 18:58:21,839 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/385be4656c4d4510a6cd18652e6ee7af as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/385be4656c4d4510a6cd18652e6ee7af 2023-04-06 18:58:21,842 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into 385be4656c4d4510a6cd18652e6ee7af(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:21,842 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:21,842 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=12, startTime=1680807501424; duration=0sec 2023-04-06 18:58:21,842 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:21,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7ae1cd31ddf74761a7ac04875e26d1c9 2023-04-06 18:58:21,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807561903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680807501886, max timestamp=1680807501903Num uniques 1113; 2023-04-06 18:58:21,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-06 18:58:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680807501886, max timestamp=1680807501903Num uniques 1113; 2023-04-06 18:58:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-06 18:58:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680807501886, max timestamp=1680807501903Num uniques 1113; 2023-04-06 18:58:21,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-06 18:58:21,925 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:21,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:21,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:21,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:21,928 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:21,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:21,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:21,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,928 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,929 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,929 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:21,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:21,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:21,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:21,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:21,930 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:21,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:21,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:21,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:21,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3157d0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:21,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:21,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:21,935 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:21,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,935 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:21,936 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:21,936 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,936 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,936 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:21,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:21,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:21,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:21,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:21,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:21,938 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:21,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:21,939 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,940 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,941 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:21,944 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:21,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:21,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:21,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:21,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:21,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:21,944 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:21,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:21,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:21,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:21,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:21,945 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:21,945 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:21,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807501951, max timestamp=1680807501969Num uniques 1072; 2023-04-06 18:58:21,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 18:58:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807501951, max timestamp=1680807501969Num uniques 1072; 2023-04-06 18:58:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 18:58:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680807501951, max timestamp=1680807501969Num uniques 1072; 2023-04-06 18:58:21,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807501983, max timestamp=1680807502018Num uniques 1081; 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807501983, max timestamp=1680807502018Num uniques 1081; 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680807501983, max timestamp=1680807502018Num uniques 1081; 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:58:22,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 18:58:22,045 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:22,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:22,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:22,047 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:22,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:22,047 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:22,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,048 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,049 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,049 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:22,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:22,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:22,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:22,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,049 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:22,050 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,050 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:22,050 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:22,050 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:22,050 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:22,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,051 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,051 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,051 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:22,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:22,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70b3fd72[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:22,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,052 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,052 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:22,052 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:22,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:22,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,053 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,053 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:22,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:22,055 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:22,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:22,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:22,056 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:22,056 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:22,057 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,057 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,057 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,057 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,058 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,060 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:22,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:22,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:22,060 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,060 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:22,060 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:22,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807502031, max timestamp=1680807502061Num uniques 1090; 2023-04-06 18:58:22,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:58:22,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807502031, max timestamp=1680807502061Num uniques 1090; 2023-04-06 18:58:22,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:58:22,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680807502031, max timestamp=1680807502061Num uniques 1090; 2023-04-06 18:58:22,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-06 18:58:22,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680807502077, max timestamp=1680807502105Num uniques 1049; 2023-04-06 18:58:22,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-06 18:58:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680807502077, max timestamp=1680807502105Num uniques 1049; 2023-04-06 18:58:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-06 18:58:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680807502077, max timestamp=1680807502105Num uniques 1049; 2023-04-06 18:58:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-06 18:58:22,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807562141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807562146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807562147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,160 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:22,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:22,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:22,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:22,163 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:22,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:22,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:22,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,163 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:22,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:22,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:22,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:22,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:22,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:22,165 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:22,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,166 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,167 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1180c5bc[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:22,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:22,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,170 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:22,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:22,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:22,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,171 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:22,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:22,173 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:22,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:22,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:22,173 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:22,173 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:22,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,174 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,176 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:22,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:22,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,178 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:22,179 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:22,179 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:22,178 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,179 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,179 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,179 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,179 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,179 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807562252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807562258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807562258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,279 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:22,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/21af32d511184337b719956dc9d1a9be 2023-04-06 18:58:22,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:22,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:22,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:22,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:22,281 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:22,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:22,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:22,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:22,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:22,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:22,284 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,284 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:22,284 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:22,284 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:22,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,285 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,285 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,285 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef0d294[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:22,287 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:22,287 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/20779dce369047da8797e6341372a114 2023-04-06 18:58:22,287 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,287 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:22,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:22,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:22,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,288 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:22,288 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,288 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7ae1cd31ddf74761a7ac04875e26d1c9 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9 2023-04-06 18:58:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:22,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:22,291 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:22,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:22,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:22,291 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:22,291 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:22,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9, entries=250, sequenceid=327, filesize=16.7 K 2023-04-06 18:58:22,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/21af32d511184337b719956dc9d1a9be as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be 2023-04-06 18:58:22,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be, entries=150, sequenceid=327, filesize=11.9 K 2023-04-06 18:58:22,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/20779dce369047da8797e6341372a114 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114 2023-04-06 18:58:22,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114, entries=150, sequenceid=327, filesize=11.9 K 2023-04-06 18:58:22,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=154.31 KB/158010 for 50e5e6c3a16e85ee7fcdf95128e439dc in 839ms, sequenceid=327, compaction requested=true 2023-04-06 18:58:22,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:22,301 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:22,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44356 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:22,302 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:22,302 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,302 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,302 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2fd2654a60f24e4b880aaa51bf6dd709, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=43.3 K 2023-04-06 18:58:22,303 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b099ad733bf44f4ca0885b39454a1243, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=36.2 K 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2fd2654a60f24e4b880aaa51bf6dd709, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680807500899 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b099ad733bf44f4ca0885b39454a1243, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680807500899 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2434ea40d3b2460abf25fc556b5ba75b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680807500929 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4c1bc48e5ada4e04b5e59ec37a78bc18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680807500929 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7ae1cd31ddf74761a7ac04875e26d1c9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680807501119 2023-04-06 18:58:22,303 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 21af32d511184337b719956dc9d1a9be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680807501119 2023-04-06 18:58:22,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:22,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:22,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:22,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,309 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#445 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:22,309 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:22,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:22,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:22,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,314 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:22,314 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:22,313 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:22,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:22,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:22,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:22,414 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:22,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:22,417 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:22,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:22,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:22,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,418 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,418 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:22,418 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:22,418 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,419 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:22,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:22,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:22,420 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:22,420 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:22,420 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=154.31 KB heapSize=431.63 KB 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:22,420 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:22,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:22,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807562466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807562466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807562466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807562469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807562569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807562569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807562577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,722 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/d11319d228f04fd09ada3f99dc1f788d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/d11319d228f04fd09ada3f99dc1f788d 2023-04-06 18:58:22,726 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into d11319d228f04fd09ada3f99dc1f788d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:22,726 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/44459d9c5efe48f2a9cb72dbd2b5ecce as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/44459d9c5efe48f2a9cb72dbd2b5ecce 2023-04-06 18:58:22,726 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:22,726 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=13, startTime=1680807502301; duration=0sec 2023-04-06 18:58:22,726 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:22,727 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-06 18:58:22,727 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-06 18:58:22,727 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-06 18:58:22,727 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. because compaction request was cancelled 2023-04-06 18:58:22,729 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into 44459d9c5efe48f2a9cb72dbd2b5ecce(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:22,729 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:22,730 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=13, startTime=1680807502302; duration=0sec 2023-04-06 18:58:22,730 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807562770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57168 deadline: 1680807562771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,777 DEBUG [Thread-3117] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8219 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:58:22,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807562779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807562780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:22,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807562781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:22,827 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/60ac866b8b5446bbb648a60c519ef342 2023-04-06 18:58:23,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807563088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:57132 deadline: 1680807563089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807563089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,234 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d020c019d505479ab9c1e56aba126cdb 2023-04-06 18:58:23,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807563279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807563594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807563595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:23,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807563597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,642 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/8c119061ead346b2880076e8d5646777 2023-04-06 18:58:23,645 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/60ac866b8b5446bbb648a60c519ef342 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342 2023-04-06 18:58:23,647 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342, entries=150, sequenceid=537, filesize=11.9 K 2023-04-06 18:58:23,648 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d020c019d505479ab9c1e56aba126cdb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb 2023-04-06 18:58:23,650 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb, entries=150, sequenceid=537, filesize=11.9 K 2023-04-06 18:58:23,651 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/8c119061ead346b2880076e8d5646777 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777 2023-04-06 18:58:23,654 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777, entries=150, sequenceid=537, filesize=11.9 K 2023-04-06 18:58:23,654 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.58 KB/441936, currentSize=26.84 KB/27480 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1234ms, sequenceid=537, compaction requested=true 2023-04-06 18:58:23,654 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:23,654 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:23,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:23,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:23,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:23,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:23,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,655 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:23,655 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,656 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:23,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:23,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b8a854d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:23,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:23,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:23,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:23,658 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:23,658 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:23,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:23,660 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:23,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:23,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:23,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:23,661 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:23,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:23,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:23,662 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,663 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,664 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:23,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:23,665 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:23,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:23,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:23,665 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,665 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:23,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:23,665 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,765 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:23,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:23,768 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:23,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:23,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:23,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:23,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,768 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:23,886 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:23,886 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:23,886 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:23,886 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:23,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:23,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:23,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:23,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:23,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:23,888 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:23,888 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:23,888 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:23,888 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:23,888 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:58:23,888 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:23,888 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:23,888 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:23,889 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:23,889 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:23,889 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:24,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:24,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:24,300 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/6831ec86019d4aecac4e26b2698cac73 2023-04-06 18:58:24,306 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/bbb7c4bcc1cc499e947752ef8a93017a 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807504332, max timestamp=1680807504345Num uniques 859; 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807504332, max timestamp=1680807504345Num uniques 859; 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807504332, max timestamp=1680807504345Num uniques 859; 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:24,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807504392, max timestamp=1680807504415Num uniques 1018; 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807504392, max timestamp=1680807504415Num uniques 1018; 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807504392, max timestamp=1680807504415Num uniques 1018; 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:24,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807504475, max timestamp=1680807504515Num uniques 1027; 2023-04-06 18:58:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807504475, max timestamp=1680807504515Num uniques 1027; 2023-04-06 18:58:24,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:24,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680807504475, max timestamp=1680807504515Num uniques 1027; 2023-04-06 18:58:24,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 18:58:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807504585, max timestamp=1680807504608Num uniques 1036; 2023-04-06 18:58:24,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 18:58:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807504585, max timestamp=1680807504608Num uniques 1036; 2023-04-06 18:58:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680807504585, max timestamp=1680807504608Num uniques 1036; 2023-04-06 18:58:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 18:58:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 18:58:24,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680807504618, max timestamp=1680807504661Num uniques 1095; 2023-04-06 18:58:24,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 18:58:24,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680807504618, max timestamp=1680807504661Num uniques 1095; 2023-04-06 18:58:24,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 18:58:24,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680807504618, max timestamp=1680807504661Num uniques 1095; 2023-04-06 18:58:24,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 18:58:24,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807564697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807564698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807564699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807564704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,714 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/78fd7346f3fe41ee8638c2298308aaed 2023-04-06 18:58:24,717 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/6831ec86019d4aecac4e26b2698cac73 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73 2023-04-06 18:58:24,719 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73, entries=100, sequenceid=546, filesize=9.5 K 2023-04-06 18:58:24,720 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/bbb7c4bcc1cc499e947752ef8a93017a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a 2023-04-06 18:58:24,722 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a, entries=100, sequenceid=546, filesize=9.5 K 2023-04-06 18:58:24,723 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/78fd7346f3fe41ee8638c2298308aaed as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed 2023-04-06 18:58:24,725 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed, entries=100, sequenceid=546, filesize=9.5 K 2023-04-06 18:58:24,726 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 50e5e6c3a16e85ee7fcdf95128e439dc in 838ms, sequenceid=546, compaction requested=true 2023-04-06 18:58:24,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:24,726 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:24,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:24,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:24,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:24,726 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:24,727 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,727 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,727 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:24,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:24,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64bda123[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:24,728 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,729 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:24,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:24,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:24,729 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,729 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:24,729 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:24,730 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,729 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,730 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:24,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:24,731 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:24,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:24,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:24,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:24,731 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:24,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,732 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,732 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,733 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:24,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:24,736 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,736 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:24,736 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,736 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:24,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=167.72 KB heapSize=467.86 KB 2023-04-06 18:58:24,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680807504682, max timestamp=1680807504810Num uniques 1204; 2023-04-06 18:58:24,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:24,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 18:58:24,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/ecf50f9f94e2412688a65c7b3e9fe039 2023-04-06 18:58:24,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807564822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807564823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807564829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807564830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,836 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:24,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:24,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:24,839 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:24,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:24,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:24,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:24,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,840 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:24,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:24,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:24,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:24,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:24,841 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:24,842 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,842 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,850 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:24,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:24,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:24,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7307352d[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:24,851 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,863 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:24,863 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,864 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:24,863 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:24,864 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:24,864 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:24,864 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,864 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,864 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:24,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:24,866 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:24,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:24,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:24,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:24,867 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:24,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,867 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,868 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,869 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,870 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:24,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:24,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:24,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:24,870 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:24,870 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,870 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:24,870 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,871 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:24,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:24,870 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807564931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807564931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807564937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807564937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,971 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:24,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:24,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:24,973 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:24,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:24,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:24,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:24,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,973 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,974 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:24,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:24,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:24,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:24,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:24,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:24,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:24,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:24,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:24,976 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,976 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,977 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:24,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:24,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9b3dab[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:24,978 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,979 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:24,979 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,979 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:24,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:24,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:24,980 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,980 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:24,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:24,982 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:24,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:24,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:24,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:24,982 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:24,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:24,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,984 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,986 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:24,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:24,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:24,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:24,986 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:24,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:24,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,986 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:24,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:24,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:24,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:24,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:24,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,087 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,089 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,089 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,090 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,090 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,091 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,091 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,092 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,092 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,092 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,092 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:25,092 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:25,093 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,093 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:25,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e47178[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:25,094 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:25,094 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,094 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:25,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,095 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:25,095 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,095 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:25,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:25,098 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:25,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:25,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:25,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:25,098 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:25,099 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,099 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,099 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,100 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:25,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,102 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:25,102 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:25,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:25,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807565135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807565135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807565141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807565141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,203 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,205 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,206 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,207 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,208 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,209 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:25,209 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:25,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,210 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,210 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:25,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:25,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d59667e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:25,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,212 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:25,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:25,212 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:25,212 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:25,213 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:25,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:25,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:25,215 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:25,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:25,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:25,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:25,215 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:25,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,217 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:25,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,219 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,219 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:25,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:25,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/2e0dfa2347174e739025d90f4406e452 2023-04-06 18:58:25,319 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,322 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,323 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,323 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,325 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,325 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,325 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,325 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:25,325 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:25,326 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:25,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:25,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173d8ea5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:25,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,328 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:25,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:25,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:25,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:25,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:25,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:25,331 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:25,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:25,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:25,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:25,331 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:25,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:25,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:25,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:25,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,338 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:25,338 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:25,438 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,440 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,441 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,442 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:25,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:25,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:25,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45999f8c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:25,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:25,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:25,446 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,446 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:25,447 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:25,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:25,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807565442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:25,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:25,449 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:25,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:25,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:25,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:25,449 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:25,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807565446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,452 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:25,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:25,452 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,452 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,453 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:25,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:25,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,453 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807565450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807565450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,553 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,558 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:25,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,559 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:25,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:25,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@320159f6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:25,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:25,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:25,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:25,562 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:25,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:25,564 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:25,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:25,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:25,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:25,564 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:25,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:25,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:25,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:25,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,569 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:25,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,569 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:25,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/f78af43e6f254caaa062e5128e875cc5 2023-04-06 18:58:25,644 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/ecf50f9f94e2412688a65c7b3e9fe039 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039 2023-04-06 18:58:25,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039, entries=250, sequenceid=654, filesize=16.7 K 2023-04-06 18:58:25,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/2e0dfa2347174e739025d90f4406e452 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452 2023-04-06 18:58:25,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452, entries=150, sequenceid=654, filesize=11.9 K 2023-04-06 18:58:25,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/f78af43e6f254caaa062e5128e875cc5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5 2023-04-06 18:58:25,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5, entries=150, sequenceid=654, filesize=11.9 K 2023-04-06 18:58:25,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~354.02 KB/362512, currentSize=53.67 KB/54960 for 50e5e6c3a16e85ee7fcdf95128e439dc in 843ms, sequenceid=654, compaction requested=true 2023-04-06 18:58:25,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:25,655 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:25,655 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:25,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:25,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:25,656 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51760 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:25,656 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:25,656 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:25,656 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:25,657 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,657 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,657 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/d11319d228f04fd09ada3f99dc1f788d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=50.5 K 2023-04-06 18:58:25,657 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/44459d9c5efe48f2a9cb72dbd2b5ecce, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=45.8 K 2023-04-06 18:58:25,657 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d11319d228f04fd09ada3f99dc1f788d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680807501119 2023-04-06 18:58:25,657 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 44459d9c5efe48f2a9cb72dbd2b5ecce, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680807501119 2023-04-06 18:58:25,657 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 60ac866b8b5446bbb648a60c519ef342, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680807502133 2023-04-06 18:58:25,657 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d020c019d505479ab9c1e56aba126cdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680807502133 2023-04-06 18:58:25,657 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6831ec86019d4aecac4e26b2698cac73, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1680807502465 2023-04-06 18:58:25,658 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting bbb7c4bcc1cc499e947752ef8a93017a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1680807502465 2023-04-06 18:58:25,658 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting ecf50f9f94e2412688a65c7b3e9fe039, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504682 2023-04-06 18:58:25,658 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2e0dfa2347174e739025d90f4406e452, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504703 2023-04-06 18:58:25,663 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:25,663 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:25,669 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:25,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:25,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:25,680 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:25,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:25,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:25,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:25,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:25,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:25,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:25,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:25,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:25,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:25,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:25,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:25,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:25,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:25,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:25,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:25,683 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:25,683 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:25,686 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/1a228ca0446f411ca65c3569ba8309f6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1a228ca0446f411ca65c3569ba8309f6 2023-04-06 18:58:25,690 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into 1a228ca0446f411ca65c3569ba8309f6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:25,690 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:25,690 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=12, startTime=1680807505655; duration=0sec 2023-04-06 18:58:25,690 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:25,690 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:25,691 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:25,691 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:25,691 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:25,692 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/385be4656c4d4510a6cd18652e6ee7af, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=57.6 K 2023-04-06 18:58:25,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 385be4656c4d4510a6cd18652e6ee7af, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680807500929 2023-04-06 18:58:25,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 20779dce369047da8797e6341372a114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680807501119 2023-04-06 18:58:25,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8c119061ead346b2880076e8d5646777, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680807502133 2023-04-06 18:58:25,694 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 78fd7346f3fe41ee8638c2298308aaed, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1680807502465 2023-04-06 18:58:25,694 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f78af43e6f254caaa062e5128e875cc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504703 2023-04-06 18:58:25,700 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:25,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:25,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807505974, max timestamp=1680807505980Num uniques 859; 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807505974, max timestamp=1680807505980Num uniques 859; 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807505974, max timestamp=1680807505980Num uniques 859; 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:25,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:26,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807565997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807566001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807566001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807566002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,071 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7331b175f08b41348820f69f74b7c3dd as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7331b175f08b41348820f69f74b7c3dd 2023-04-06 18:58:26,074 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into 7331b175f08b41348820f69f74b7c3dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:26,074 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:26,074 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=12, startTime=1680807505655; duration=0sec 2023-04-06 18:58:26,075 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:26,100 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/85059efd6cbd4eaba6f95c0f1713f6d3 2023-04-06 18:58:26,108 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/7298fe9f423943ea88d800832c063236 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7298fe9f423943ea88d800832c063236 2023-04-06 18:58:26,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807566108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,111 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into 7298fe9f423943ea88d800832c063236(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:26,111 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:26,111 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=11, startTime=1680807505656; duration=0sec 2023-04-06 18:58:26,112 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:26,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807566109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807566114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807566114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807566312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807566318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807566321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807566323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,508 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/24f6e3ac72414d179b34c7c02ee04a0b 2023-04-06 18:58:26,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807566618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807566623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807566628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:26,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807566628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,916 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/e91435ab3c614bd18a7d0994484dd208 2023-04-06 18:58:26,919 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/85059efd6cbd4eaba6f95c0f1713f6d3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3 2023-04-06 18:58:26,922 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3, entries=100, sequenceid=665, filesize=9.5 K 2023-04-06 18:58:26,923 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/24f6e3ac72414d179b34c7c02ee04a0b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b 2023-04-06 18:58:26,925 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b, entries=100, sequenceid=665, filesize=9.5 K 2023-04-06 18:58:26,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/e91435ab3c614bd18a7d0994484dd208 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208 2023-04-06 18:58:26,929 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208, entries=100, sequenceid=665, filesize=9.5 K 2023-04-06 18:58:26,929 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1246ms, sequenceid=665, compaction requested=false 2023-04-06 18:58:26,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:26,929 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:26,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:26,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:26,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,929 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,930 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:26,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247da35e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:26,931 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:26,931 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,932 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:26,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,932 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:26,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:26,932 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:26,933 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:26,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:26,933 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:26,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:26,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:26,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:26,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:26,934 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:26,934 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:26,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:26,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,935 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:26,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:26,936 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,937 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:26,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:26,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:26,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:26,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:26,938 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:26,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:26,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:26,938 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:26,938 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:26,938 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,038 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:27,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:27,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:27,041 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:27,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:27,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:27,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:27,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,041 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,042 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:27,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:27,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,042 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:27,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:27,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:27,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:27,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:27,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:27,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:27,043 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:27,043 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:27,043 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:27,043 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:27,044 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:27,127 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:27,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807567136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807567140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807567140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807567140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807567241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807567241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807567245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807567247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807567446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807567447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807567448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,450 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/78c9688a1a844576a4d34b0cb5381ff8 2023-04-06 18:58:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807567451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,458 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d11f59a96d864e93924699191a99cdfa 2023-04-06 18:58:27,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807567750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807567752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807567752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:27,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807567758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,866 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/430319c35b95493289fdd952cc85d063 2023-04-06 18:58:27,869 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/78c9688a1a844576a4d34b0cb5381ff8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8 2023-04-06 18:58:27,872 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8, entries=150, sequenceid=702, filesize=11.9 K 2023-04-06 18:58:27,872 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/d11f59a96d864e93924699191a99cdfa as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa 2023-04-06 18:58:27,875 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa, entries=150, sequenceid=702, filesize=11.9 K 2023-04-06 18:58:27,875 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/430319c35b95493289fdd952cc85d063 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063 2023-04-06 18:58:27,877 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063, entries=150, sequenceid=702, filesize=11.9 K 2023-04-06 18:58:27,878 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 50e5e6c3a16e85ee7fcdf95128e439dc in 835ms, sequenceid=702, compaction requested=true 2023-04-06 18:58:27,878 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:27,878 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:27,879 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:27,879 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:27,880 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,880 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:27,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:27,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fce49d2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:27,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:27,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,881 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:27,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:27,881 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:27,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,881 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:27,882 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:27,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:27,883 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:27,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:27,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:27,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:27,883 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:27,884 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,884 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:27,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,884 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:27,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:27,885 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,886 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,887 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:27,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:27,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:27,887 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,887 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:27,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,987 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:27,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:27,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:27,990 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:27,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:27,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:27,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:27,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,991 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:27,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:27,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:27,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:27,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:27,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:27,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:27,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:27,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:27,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:27,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:27,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:27,993 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:27,993 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:27,993 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:27,994 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:28,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:28,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:28,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807568281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807508276, max timestamp=1680807508285Num uniques 859; 2023-04-06 18:58:28,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:28,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807508276, max timestamp=1680807508285Num uniques 859; 2023-04-06 18:58:28,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:28,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807508276, max timestamp=1680807508285Num uniques 859; 2023-04-06 18:58:28,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:28,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807568316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807568319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807568322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807568388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,399 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2885b14871924079b07d35667a353e71 2023-04-06 18:58:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807568428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807568428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807568429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807568594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807568634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807568634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807568634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,807 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/62de1afceb0040e393ec61d535bc4c85 2023-04-06 18:58:28,814 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=713 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/6cf2b06d46fb4f3fb759f96768a5a92f 2023-04-06 18:58:28,816 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2885b14871924079b07d35667a353e71 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71 2023-04-06 18:58:28,819 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71, entries=150, sequenceid=713, filesize=11.9 K 2023-04-06 18:58:28,819 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/62de1afceb0040e393ec61d535bc4c85 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85 2023-04-06 18:58:28,822 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85, entries=150, sequenceid=713, filesize=11.9 K 2023-04-06 18:58:28,822 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/6cf2b06d46fb4f3fb759f96768a5a92f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f 2023-04-06 18:58:28,825 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f, entries=150, sequenceid=713, filesize=11.9 K 2023-04-06 18:58:28,825 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 50e5e6c3a16e85ee7fcdf95128e439dc in 832ms, sequenceid=713, compaction requested=true 2023-04-06 18:58:28,825 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:28,825 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:28,826 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,826 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,827 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,827 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:28,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a2c858[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:28,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:28,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,828 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:28,828 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:28,828 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,829 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:28,829 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:28,830 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:28,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:28,832 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:28,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:28,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:28,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:28,832 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:28,833 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,833 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,833 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,833 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,834 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:28,839 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:28,839 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:28,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:28,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:28,840 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:28,840 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:28,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:28,910 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:28,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:28,940 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:28,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:28,942 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:28,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:28,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:28,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:28,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,942 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,943 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:28,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:28,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,943 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:28,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:28,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:28,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:28,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:28,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:28,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:28,944 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:28,944 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:28,944 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:28,945 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:28,945 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:28,945 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,945 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,946 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,946 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e54f593[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:28,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:28,947 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,948 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:28,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:28,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,948 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:28,948 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:28,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807568940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:28,949 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:28,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:28,952 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:28,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:28,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:28,952 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:28,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,952 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807568941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807568942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:28,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,953 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:28,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:28,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807568949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:28,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:28,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:28,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:28,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:28,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,964 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:28,964 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:28,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807569053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,064 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807569062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,068 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,068 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,069 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,069 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,070 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,071 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,074 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,074 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,074 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,074 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,074 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,075 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,075 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fadc54[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,076 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,077 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,077 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,077 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,078 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,078 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,079 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,079 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,079 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,080 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,081 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,082 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,083 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:29,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,083 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,083 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,083 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,084 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,184 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,186 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,186 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,187 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,187 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,188 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,189 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,189 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,190 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,190 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad781a0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,191 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,191 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,191 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,192 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,192 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,194 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,194 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,194 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,195 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,195 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,195 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,196 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,198 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,199 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,199 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,199 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,199 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807569258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807569269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,299 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,302 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,302 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,303 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,303 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,304 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,304 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,304 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,304 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,304 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af778aa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,306 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,307 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,310 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,310 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,311 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,311 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,312 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,312 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,313 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,314 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/6293e1fa0e8145c5be4315d428ae3d50 2023-04-06 18:58:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,316 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,317 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,318 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,322 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,322 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,322 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,422 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,425 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,427 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,431 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,431 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,431 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,432 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,432 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,438 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,438 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,439 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@346e40a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,440 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,447 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,455 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,455 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,457 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,457 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57206 deadline: 1680807569451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57132 deadline: 1680807569455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,463 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,464 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,465 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,466 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:29,466 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,467 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,467 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,467 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,567 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,570 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57182 deadline: 1680807569565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,571 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,572 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,572 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,572 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,572 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,573 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,573 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,574 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,574 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,574 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72af804a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,576 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,576 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,576 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,577 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,577 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:29,579 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,579 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:57130 deadline: 1680807569574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,579 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,579 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,579 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,580 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,580 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,580 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,581 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,582 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,584 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,584 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,584 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,584 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,684 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:29,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:29,687 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:29,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:29,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:29,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:29,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:29,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:29,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:29,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:29,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:29,691 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:29,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. started... 2023-04-06 18:58:29,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. as already flushing 2023-04-06 18:58:29,692 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,692 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:29,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:29,693 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,693 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,694 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:29,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4443c53e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:29,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:29,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,695 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:29,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:29,695 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,696 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:29,695 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:29,696 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:29,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,696 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:29,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:29,698 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:29,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:29,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:29,698 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:29,698 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:29,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:29,699 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,700 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:29,700 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:29,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:29,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:29,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:29,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,703 DEBUG [Thread-3123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:29,703 DEBUG [Thread-3123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:29,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:29,703 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:29,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/115f6c5a90d246f58621c7c0b04e8cda 2023-04-06 18:58:29,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/e2a92791751141dab950d7def17c18f8 2023-04-06 18:58:29,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/6293e1fa0e8145c5be4315d428ae3d50 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50 2023-04-06 18:58:29,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50, entries=200, sequenceid=748, filesize=14.3 K 2023-04-06 18:58:29,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/115f6c5a90d246f58621c7c0b04e8cda as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda 2023-04-06 18:58:29,741 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda, entries=150, sequenceid=748, filesize=11.9 K 2023-04-06 18:58:29,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/e2a92791751141dab950d7def17c18f8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8 2023-04-06 18:58:29,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8, entries=150, sequenceid=748, filesize=11.9 K 2023-04-06 18:58:29,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 50e5e6c3a16e85ee7fcdf95128e439dc in 834ms, sequenceid=748, compaction requested=true 2023-04-06 18:58:29,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:29,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:29,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:29,744 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:29,744 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:29,744 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:29,745 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:29,745 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/B is initiating minor compaction (all files) 2023-04-06 18:58:29,745 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/B in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,745 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1a228ca0446f411ca65c3569ba8309f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=57.8 K 2023-04-06 18:58:29,746 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61642 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:29,746 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/A is initiating minor compaction (all files) 2023-04-06 18:58:29,746 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/A in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:29,746 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7331b175f08b41348820f69f74b7c3dd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=60.2 K 2023-04-06 18:58:29,746 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 1a228ca0446f411ca65c3569ba8309f6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504703 2023-04-06 18:58:29,746 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7331b175f08b41348820f69f74b7c3dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504703 2023-04-06 18:58:29,746 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 24f6e3ac72414d179b34c7c02ee04a0b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680807504827 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 85059efd6cbd4eaba6f95c0f1713f6d3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680807504827 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d11f59a96d864e93924699191a99cdfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680807506000 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 78c9688a1a844576a4d34b0cb5381ff8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680807506000 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 62de1afceb0040e393ec61d535bc4c85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680807507133 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2885b14871924079b07d35667a353e71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680807507133 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 115f6c5a90d246f58621c7c0b04e8cda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680807508315 2023-04-06 18:58:29,747 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6293e1fa0e8145c5be4315d428ae3d50, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680807508315 2023-04-06 18:58:29,760 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#A#compaction#471 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:29,764 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#B#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:29,768 DEBUG [Thread-3130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d3df389 to 127.0.0.1:63695 2023-04-06 18:58:29,768 DEBUG [Thread-3130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:29,770 DEBUG [Thread-3132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f69c3f to 127.0.0.1:63695 2023-04-06 18:58:29,770 DEBUG [Thread-3132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:29,772 DEBUG [Thread-3124] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6523d347 to 127.0.0.1:63695 2023-04-06 18:58:29,772 DEBUG [Thread-3124] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:29,773 DEBUG [Thread-3126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28072326 to 127.0.0.1:63695 2023-04-06 18:58:29,773 DEBUG [Thread-3126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:29,775 DEBUG [Thread-3128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d90797d to 127.0.0.1:63695 2023-04-06 18:58:29,775 DEBUG [Thread-3128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:29,803 DEBUG [Thread-3123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:29,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:30,078 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 18:58:30,078 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:30,078 DEBUG [Thread-3119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d7deb76 to 127.0.0.1:63695 2023-04-06 18:58:30,079 DEBUG [Thread-3119] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:30,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:30,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:30,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:30,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:30,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:30,084 DEBUG [Thread-3121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:63695 2023-04-06 18:58:30,084 DEBUG [Thread-3121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:30,187 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/6e4713baf825455f8bc537a5c0b9b85b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6e4713baf825455f8bc537a5c0b9b85b 2023-04-06 18:58:30,188 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/30d8f83208e347a49c9cc6cc4c1876c4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/30d8f83208e347a49c9cc6cc4c1876c4 2023-04-06 18:58:30,192 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/A of 50e5e6c3a16e85ee7fcdf95128e439dc into 6e4713baf825455f8bc537a5c0b9b85b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:30,192 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:30,192 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/A, priority=11, startTime=1680807509744; duration=0sec 2023-04-06 18:58:30,193 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:30,193 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/B of 50e5e6c3a16e85ee7fcdf95128e439dc into 30d8f83208e347a49c9cc6cc4c1876c4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:30,193 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:30,193 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:30,193 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/B, priority=11, startTime=1680807509744; duration=0sec 2023-04-06 18:58:30,193 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:30,194 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59168 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:30,194 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 50e5e6c3a16e85ee7fcdf95128e439dc/C is initiating minor compaction (all files) 2023-04-06 18:58:30,194 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 50e5e6c3a16e85ee7fcdf95128e439dc/C in TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:30,195 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7298fe9f423943ea88d800832c063236, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp, totalSize=57.8 K 2023-04-06 18:58:30,195 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 7298fe9f423943ea88d800832c063236, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680807504703 2023-04-06 18:58:30,195 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e91435ab3c614bd18a7d0994484dd208, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1680807504827 2023-04-06 18:58:30,196 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 430319c35b95493289fdd952cc85d063, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680807506000 2023-04-06 18:58:30,196 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6cf2b06d46fb4f3fb759f96768a5a92f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=713, earliestPutTs=1680807507133 2023-04-06 18:58:30,196 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e2a92791751141dab950d7def17c18f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680807508315 2023-04-06 18:58:30,205 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 50e5e6c3a16e85ee7fcdf95128e439dc#C#compaction#474 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:30,464 DEBUG [Thread-3115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50d6899d to 127.0.0.1:63695 2023-04-06 18:58:30,464 DEBUG [Thread-3115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:30,471 DEBUG [Thread-3113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x682c0b9e to 127.0.0.1:63695 2023-04-06 18:58:30,471 DEBUG [Thread-3113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:30,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7e8e8b95d5f748159fcae8d7cdd4c213 2023-04-06 18:58:30,616 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/1ec69440c2314c53aca81a7cb8a8e17c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1ec69440c2314c53aca81a7cb8a8e17c 2023-04-06 18:58:30,621 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 50e5e6c3a16e85ee7fcdf95128e439dc/C of 50e5e6c3a16e85ee7fcdf95128e439dc into 1ec69440c2314c53aca81a7cb8a8e17c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:30,621 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:30,621 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc., storeName=50e5e6c3a16e85ee7fcdf95128e439dc/C, priority=11, startTime=1680807509744; duration=0sec 2023-04-06 18:58:30,621 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:30,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/f1dd676f046a41249797576a03d52ead 2023-04-06 18:58:31,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/37575d71cece46419435949f7b62faf1 2023-04-06 18:58:31,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/7e8e8b95d5f748159fcae8d7cdd4c213 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7e8e8b95d5f748159fcae8d7cdd4c213 2023-04-06 18:58:31,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7e8e8b95d5f748159fcae8d7cdd4c213, entries=150, sequenceid=759, filesize=11.9 K 2023-04-06 18:58:31,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/f1dd676f046a41249797576a03d52ead as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/f1dd676f046a41249797576a03d52ead 2023-04-06 18:58:31,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/f1dd676f046a41249797576a03d52ead, entries=150, sequenceid=759, filesize=11.9 K 2023-04-06 18:58:31,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/37575d71cece46419435949f7b62faf1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/37575d71cece46419435949f7b62faf1 2023-04-06 18:58:31,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/37575d71cece46419435949f7b62faf1, entries=150, sequenceid=759, filesize=11.9 K 2023-04-06 18:58:31,335 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=20.13 KB/20610 for 50e5e6c3a16e85ee7fcdf95128e439dc in 1257ms, sequenceid=759, compaction requested=false 2023-04-06 18:58:31,335 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:32,849 DEBUG [Thread-3117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6662e946 to 127.0.0.1:63695 2023-04-06 18:58:32,850 DEBUG [Thread-3117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:32,850 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:58:32,850 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 229 2023-04-06 18:58:32,851 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-06 18:58:32,851 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-06 18:58:32,851 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-06 18:58:32,851 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 204 2023-04-06 18:58:32,851 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2067 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 6201 rows 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2071 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 6210 rows 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2070 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 6204 rows 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2055 2023-04-06 18:58:32,852 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 6165 rows 2023-04-06 18:58:32,853 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(402): scanned 2060 2023-04-06 18:58:32,853 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(403): verified 6180 rows 2023-04-06 18:58:32,853 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:58:32,853 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a7ac4a1 to 127.0.0.1:63695 2023-04-06 18:58:32,853 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:32,855 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:58:32,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:58:32,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:32,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:32,861 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807512861"}]},"ts":"1680807512861"} 2023-04-06 18:58:32,862 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:58:32,872 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:58:32,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, UNASSIGN}] 2023-04-06 18:58:32,873 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, UNASSIGN 2023-04-06 18:58:32,874 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=50e5e6c3a16e85ee7fcdf95128e439dc, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:32,874 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807512874"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807512874"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807512874"}]},"ts":"1680807512874"} 2023-04-06 18:58:32,875 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:32,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:33,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:33,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 50e5e6c3a16e85ee7fcdf95128e439dc, disabling compactions & flushes 2023-04-06 18:58:33,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:33,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. after waiting 0 ms 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:33,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 50e5e6c3a16e85ee7fcdf95128e439dc 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=A 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=B 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 50e5e6c3a16e85ee7fcdf95128e439dc, store=C 2023-04-06 18:58:33,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:33,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2148414b478247e89d30d9983cf13bd3 2023-04-06 18:58:33,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:33,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/1da413f21074427cb1c4229fe7edfac2 2023-04-06 18:58:33,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:33,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=769 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/459a652cfee04a908da2d03b50dd0f7b 2023-04-06 18:58:33,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/A/2148414b478247e89d30d9983cf13bd3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2148414b478247e89d30d9983cf13bd3 2023-04-06 18:58:33,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2148414b478247e89d30d9983cf13bd3, entries=150, sequenceid=769, filesize=11.9 K 2023-04-06 18:58:33,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/B/1da413f21074427cb1c4229fe7edfac2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1da413f21074427cb1c4229fe7edfac2 2023-04-06 18:58:33,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1da413f21074427cb1c4229fe7edfac2, entries=150, sequenceid=769, filesize=11.9 K 2023-04-06 18:58:33,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/.tmp/C/459a652cfee04a908da2d03b50dd0f7b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/459a652cfee04a908da2d03b50dd0f7b 2023-04-06 18:58:33,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/459a652cfee04a908da2d03b50dd0f7b, entries=150, sequenceid=769, filesize=11.9 K 2023-04-06 18:58:33,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 50e5e6c3a16e85ee7fcdf95128e439dc in 875ms, sequenceid=769, compaction requested=true 2023-04-06 18:58:33,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/5a76aef8ce2846fe9b26a0c32581fbb7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ca8062e8a1824f31a007ba7ff31c0af5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a7a6e727560e4508b95ce91bd9132d01, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/f7838526dfc5486da707428dc1656c25, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2fd2654a60f24e4b880aaa51bf6dd709, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/d11319d228f04fd09ada3f99dc1f788d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7331b175f08b41348820f69f74b7c3dd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50] to archive 2023-04-06 18:58:33,926 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:33,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/5a76aef8ce2846fe9b26a0c32581fbb7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/5a76aef8ce2846fe9b26a0c32581fbb7 2023-04-06 18:58:33,928 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/9f8797fabf7041f7ad4c4412ad0ddbaf 2023-04-06 18:58:33,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7af6e651ea244511a9b014e24f1755fd 2023-04-06 18:58:33,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ab1b890bfec84a1d90920e12a0dbe65b 2023-04-06 18:58:33,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ca8062e8a1824f31a007ba7ff31c0af5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ca8062e8a1824f31a007ba7ff31c0af5 2023-04-06 18:58:33,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/1163ab436240451e8dd03f4b1a766d1f 2023-04-06 18:58:33,932 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/e631cfa391654626946cb22d0cccb005 2023-04-06 18:58:33,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/4fc708b0f66545b7aa625aba039ebb1b 2023-04-06 18:58:33,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a7a6e727560e4508b95ce91bd9132d01 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a7a6e727560e4508b95ce91bd9132d01 2023-04-06 18:58:33,934 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/c177cd52943a49bfac0e5b3447ca3eba 2023-04-06 18:58:33,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/a62d03a26253438a84c48d46563a4fae 2023-04-06 18:58:33,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/f7838526dfc5486da707428dc1656c25 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/f7838526dfc5486da707428dc1656c25 2023-04-06 18:58:33,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/0c1b7bcdae6149aca2ed53a54bc4442c 2023-04-06 18:58:33,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/b5fb8a3c5e0d4d7ca0ef908bd92f1502 2023-04-06 18:58:33,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2fd2654a60f24e4b880aaa51bf6dd709 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2fd2654a60f24e4b880aaa51bf6dd709 2023-04-06 18:58:33,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2434ea40d3b2460abf25fc556b5ba75b 2023-04-06 18:58:33,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7ae1cd31ddf74761a7ac04875e26d1c9 2023-04-06 18:58:33,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/d11319d228f04fd09ada3f99dc1f788d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/d11319d228f04fd09ada3f99dc1f788d 2023-04-06 18:58:33,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/60ac866b8b5446bbb648a60c519ef342 2023-04-06 18:58:33,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6831ec86019d4aecac4e26b2698cac73 2023-04-06 18:58:33,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/ecf50f9f94e2412688a65c7b3e9fe039 2023-04-06 18:58:33,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7331b175f08b41348820f69f74b7c3dd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7331b175f08b41348820f69f74b7c3dd 2023-04-06 18:58:33,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/85059efd6cbd4eaba6f95c0f1713f6d3 2023-04-06 18:58:33,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/78c9688a1a844576a4d34b0cb5381ff8 2023-04-06 18:58:33,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2885b14871924079b07d35667a353e71 2023-04-06 18:58:33,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6293e1fa0e8145c5be4315d428ae3d50 2023-04-06 18:58:33,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/292d4f5437de4019b974741b1a4b98f0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d0d2fbba26d7429f8f77e94477eec13b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d684631bc138486b8a52db5da65a5d53, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/98c04e4b54de466c9b5d3dd0782dec97, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b099ad733bf44f4ca0885b39454a1243, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/44459d9c5efe48f2a9cb72dbd2b5ecce, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1a228ca0446f411ca65c3569ba8309f6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda] to archive 2023-04-06 18:58:33,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:33,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/292d4f5437de4019b974741b1a4b98f0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/292d4f5437de4019b974741b1a4b98f0 2023-04-06 18:58:33,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b997510b82be492898d79172243f6d5b 2023-04-06 18:58:33,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/a4700d78cab443b99556e84b8623fca0 2023-04-06 18:58:33,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d0d2fbba26d7429f8f77e94477eec13b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d0d2fbba26d7429f8f77e94477eec13b 2023-04-06 18:58:33,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/778df82fb77041778e30167693b7dbf9 2023-04-06 18:58:33,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/8d06b28ca09245a3becae4419777648b 2023-04-06 18:58:33,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/c03480e63575499c8efe8d82d7e16c48 2023-04-06 18:58:33,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d684631bc138486b8a52db5da65a5d53 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d684631bc138486b8a52db5da65a5d53 2023-04-06 18:58:33,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/3d5c420cf6fa496685fe30e25b103d66 2023-04-06 18:58:33,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/6d38537e3f9c435f89bc31aef15e84f1 2023-04-06 18:58:33,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/98c04e4b54de466c9b5d3dd0782dec97 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/98c04e4b54de466c9b5d3dd0782dec97 2023-04-06 18:58:33,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/550fb710de4f47dbbac04f1dbb88be03 2023-04-06 18:58:33,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/5e9940aed46042d18bafea21762107af 2023-04-06 18:58:33,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b099ad733bf44f4ca0885b39454a1243 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b099ad733bf44f4ca0885b39454a1243 2023-04-06 18:58:33,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/b03ceaab913d44cfb09de99163a5896b 2023-04-06 18:58:33,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/4c1bc48e5ada4e04b5e59ec37a78bc18 2023-04-06 18:58:33,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:33,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/44459d9c5efe48f2a9cb72dbd2b5ecce to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/44459d9c5efe48f2a9cb72dbd2b5ecce 2023-04-06 18:58:33,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/21af32d511184337b719956dc9d1a9be 2023-04-06 18:58:33,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d020c019d505479ab9c1e56aba126cdb 2023-04-06 18:58:33,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/bbb7c4bcc1cc499e947752ef8a93017a 2023-04-06 18:58:33,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1a228ca0446f411ca65c3569ba8309f6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1a228ca0446f411ca65c3569ba8309f6 2023-04-06 18:58:33,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/2e0dfa2347174e739025d90f4406e452 2023-04-06 18:58:33,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/24f6e3ac72414d179b34c7c02ee04a0b 2023-04-06 18:58:33,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/d11f59a96d864e93924699191a99cdfa 2023-04-06 18:58:33,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/62de1afceb0040e393ec61d535bc4c85 2023-04-06 18:58:33,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/115f6c5a90d246f58621c7c0b04e8cda 2023-04-06 18:58:33,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d7a796fbc38c41cb94207c785f743fac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9d32fdc7186b4d46955fbb2c20bf122e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/cc034a1c07484d82816ede13c65dc862, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/65dc7952796d41918e54a0a49c35f4b1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/385be4656c4d4510a6cd18652e6ee7af, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7298fe9f423943ea88d800832c063236, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8] to archive 2023-04-06 18:58:33,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:58:33,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d7a796fbc38c41cb94207c785f743fac to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d7a796fbc38c41cb94207c785f743fac 2023-04-06 18:58:33,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/4410e612b88b454ab6b2e6b2f6cad986 2023-04-06 18:58:33,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/d5fc468f940c4b21bade040df9973ef6 2023-04-06 18:58:33,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9d32fdc7186b4d46955fbb2c20bf122e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9d32fdc7186b4d46955fbb2c20bf122e 2023-04-06 18:58:33,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/84b70b8ef37247a4bbecf652ecd0fa1a 2023-04-06 18:58:33,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/34bcd065460f4677ab26d1d8f08f9330 2023-04-06 18:58:33,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7381ec0fea3e4526bb3dfe5ea069f264 2023-04-06 18:58:33,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/cc034a1c07484d82816ede13c65dc862 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/cc034a1c07484d82816ede13c65dc862 2023-04-06 18:58:33,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/0c9ea399e4a549b595ad345748a6febd 2023-04-06 18:58:33,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/120b19399c774e1097ef13ee3f5821cc 2023-04-06 18:58:33,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/65dc7952796d41918e54a0a49c35f4b1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/65dc7952796d41918e54a0a49c35f4b1 2023-04-06 18:58:33,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/29888e3074c3441e823bfbc0bf7e7f98 2023-04-06 18:58:33,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/35ca54fd363e4b41add740ef22073589 2023-04-06 18:58:33,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/ed5c04e7fd054bdabf7ff937479e10f7 2023-04-06 18:58:33,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/385be4656c4d4510a6cd18652e6ee7af to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/385be4656c4d4510a6cd18652e6ee7af 2023-04-06 18:58:33,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/9487e6112d4a43eb89e33b727a3b6600 2023-04-06 18:58:33,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/20779dce369047da8797e6341372a114 2023-04-06 18:58:33,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/8c119061ead346b2880076e8d5646777 2023-04-06 18:58:33,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/78fd7346f3fe41ee8638c2298308aaed 2023-04-06 18:58:33,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7298fe9f423943ea88d800832c063236 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/7298fe9f423943ea88d800832c063236 2023-04-06 18:58:33,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/f78af43e6f254caaa062e5128e875cc5 2023-04-06 18:58:33,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e91435ab3c614bd18a7d0994484dd208 2023-04-06 18:58:33,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/430319c35b95493289fdd952cc85d063 2023-04-06 18:58:33,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/6cf2b06d46fb4f3fb759f96768a5a92f 2023-04-06 18:58:33,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/e2a92791751141dab950d7def17c18f8 2023-04-06 18:58:34,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/recovered.edits/772.seqid, newMaxSeqId=772, maxSeqId=1 2023-04-06 18:58:34,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc. 2023-04-06 18:58:34,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 50e5e6c3a16e85ee7fcdf95128e439dc: 2023-04-06 18:58:34,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:34,003 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=50e5e6c3a16e85ee7fcdf95128e439dc, regionState=CLOSED 2023-04-06 18:58:34,004 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807514003"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807514003"}]},"ts":"1680807514003"} 2023-04-06 18:58:34,005 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-06 18:58:34,005 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 50e5e6c3a16e85ee7fcdf95128e439dc, server=jenkins-hbase20.apache.org,34607,1680807373848 in 1.1290 sec 2023-04-06 18:58:34,006 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-06 18:58:34,006 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=50e5e6c3a16e85ee7fcdf95128e439dc, UNASSIGN in 1.1330 sec 2023-04-06 18:58:34,007 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807514006"}]},"ts":"1680807514006"} 2023-04-06 18:58:34,007 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:58:34,008 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:58:34,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1530 sec 2023-04-06 18:58:34,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 18:58:34,969 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-06 18:58:34,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:58:34,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:34,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:34,976 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:34,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 18:58:34,984 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:34,988 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/recovered.edits] 2023-04-06 18:58:34,993 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2148414b478247e89d30d9983cf13bd3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/2148414b478247e89d30d9983cf13bd3 2023-04-06 18:58:34,996 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6e4713baf825455f8bc537a5c0b9b85b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/6e4713baf825455f8bc537a5c0b9b85b 2023-04-06 18:58:34,998 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7e8e8b95d5f748159fcae8d7cdd4c213 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/A/7e8e8b95d5f748159fcae8d7cdd4c213 2023-04-06 18:58:35,001 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1da413f21074427cb1c4229fe7edfac2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/1da413f21074427cb1c4229fe7edfac2 2023-04-06 18:58:35,003 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/30d8f83208e347a49c9cc6cc4c1876c4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/30d8f83208e347a49c9cc6cc4c1876c4 2023-04-06 18:58:35,005 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/f1dd676f046a41249797576a03d52ead to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/B/f1dd676f046a41249797576a03d52ead 2023-04-06 18:58:35,009 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1ec69440c2314c53aca81a7cb8a8e17c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/1ec69440c2314c53aca81a7cb8a8e17c 2023-04-06 18:58:35,010 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/37575d71cece46419435949f7b62faf1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/37575d71cece46419435949f7b62faf1 2023-04-06 18:58:35,012 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/459a652cfee04a908da2d03b50dd0f7b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/C/459a652cfee04a908da2d03b50dd0f7b 2023-04-06 18:58:35,015 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/recovered.edits/772.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc/recovered.edits/772.seqid 2023-04-06 18:58:35,016 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/50e5e6c3a16e85ee7fcdf95128e439dc 2023-04-06 18:58:35,016 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:58:35,019 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:35,020 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:58:35,022 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:58:35,024 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:35,024 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:58:35,024 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807515024"}]},"ts":"9223372036854775807"} 2023-04-06 18:58:35,026 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:58:35,026 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 50e5e6c3a16e85ee7fcdf95128e439dc, NAME => 'TestAcidGuarantees,,1680807487521.50e5e6c3a16e85ee7fcdf95128e439dc.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:58:35,026 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:58:35,026 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807515026"}]},"ts":"9223372036854775807"} 2023-04-06 18:58:35,027 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:58:35,028 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:35,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-06 18:58:35,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 18:58:35,078 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-06 18:58:35,098 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 228), OpenFileDescriptor=527 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=437 (was 493), ProcessCount=166 (was 166), AvailableMemoryMB=3807 (was 4184) 2023-04-06 18:58:35,110 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=527, MaxFileDescriptor=60000, SystemLoadAverage=437, ProcessCount=166, AvailableMemoryMB=3807 2023-04-06 18:58:35,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:58:35,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:58:35,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:35,114 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 18:58:35,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-06 18:58:35,115 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 18:58:35,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:35,116 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:35,116 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 empty. 2023-04-06 18:58:35,116 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:35,117 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:58:35,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:35,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:35,525 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:58:35,526 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 03155aa75aad25b899de8cf2f21dd327, NAME => 'TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp 2023-04-06 18:58:35,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:35,933 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:58:35,933 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 03155aa75aad25b899de8cf2f21dd327, disabling compactions & flushes 2023-04-06 18:58:35,934 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:35,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:35,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. after waiting 0 ms 2023-04-06 18:58:35,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:35,934 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:35,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:35,939 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 18:58:35,942 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807515942"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807515942"}]},"ts":"1680807515942"} 2023-04-06 18:58:35,944 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 18:58:35,946 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 18:58:35,946 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807515946"}]},"ts":"1680807515946"} 2023-04-06 18:58:35,948 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 18:58:35,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, ASSIGN}] 2023-04-06 18:58:35,953 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, ASSIGN 2023-04-06 18:58:35,954 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=false 2023-04-06 18:58:36,106 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:36,106 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807516106"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807516106"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807516106"}]},"ts":"1680807516106"} 2023-04-06 18:58:36,109 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:36,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:36,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:36,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 03155aa75aad25b899de8cf2f21dd327, NAME => 'TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:58:36,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:58:36,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,273 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,275 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A 2023-04-06 18:58:36,275 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A 2023-04-06 18:58:36,275 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:36,275 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName A 2023-04-06 18:58:36,276 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:36,276 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,277 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B 2023-04-06 18:58:36,277 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B 2023-04-06 18:58:36,277 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:36,277 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName B 2023-04-06 18:58:36,277 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:36,277 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,278 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C 2023-04-06 18:58:36,278 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C 2023-04-06 18:58:36,278 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:36,279 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName C 2023-04-06 18:58:36,279 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:36,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:36,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:58:36,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:36,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 18:58:36,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 03155aa75aad25b899de8cf2f21dd327; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59459693, jitterRate=-0.11398153007030487}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:58:36,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:36,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., pid=56, masterSystemTime=1680807516262 2023-04-06 18:58:36,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:36,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:36,285 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:36,286 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807516285"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807516285"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807516285"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807516285"}]},"ts":"1680807516285"} 2023-04-06 18:58:36,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-06 18:58:36,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 in 177 msec 2023-04-06 18:58:36,289 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-06 18:58:36,289 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, ASSIGN in 337 msec 2023-04-06 18:58:36,290 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 18:58:36,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807516290"}]},"ts":"1680807516290"} 2023-04-06 18:58:36,290 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 18:58:36,292 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 18:58:36,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-04-06 18:58:37,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 18:58:37,224 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-06 18:58:37,225 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x630ea023 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:37,233 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6722b207, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:37,235 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:37,236 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:37,238 DEBUG [Listener at localhost.localdomain/41219] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 18:58:37,241 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 18:58:37,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 18:58:37,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 18:58:37,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 18:58:37,654 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:58:37,657 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 18:58:37,657 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 18:58:37,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 18:58:37,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, REOPEN/MOVE}] 2023-04-06 18:58:37,666 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, REOPEN/MOVE 2023-04-06 18:58:37,667 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:37,667 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807517667"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807517667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807517667"}]},"ts":"1680807517667"} 2023-04-06 18:58:37,668 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:37,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:37,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 03155aa75aad25b899de8cf2f21dd327, disabling compactions & flushes 2023-04-06 18:58:37,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:37,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:37,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. after waiting 0 ms 2023-04-06 18:58:37,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:37,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 18:58:37,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:37,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:37,830 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 03155aa75aad25b899de8cf2f21dd327 to self. 2023-04-06 18:58:37,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:37,832 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=CLOSED 2023-04-06 18:58:37,832 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807517832"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807517832"}]},"ts":"1680807517832"} 2023-04-06 18:58:37,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-06 18:58:37,834 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 in 164 msec 2023-04-06 18:58:37,834 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34607,1680807373848; forceNewPlan=false, retain=true 2023-04-06 18:58:37,985 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:37,985 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807517985"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807517985"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807517985"}]},"ts":"1680807517985"} 2023-04-06 18:58:37,987 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:58:38,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 03155aa75aad25b899de8cf2f21dd327, NAME => 'TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.', STARTKEY => '', ENDKEY => ''} 2023-04-06 18:58:38,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 18:58:38,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,148 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,149 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A 2023-04-06 18:58:38,149 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A 2023-04-06 18:58:38,149 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:38,149 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName A 2023-04-06 18:58:38,151 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:38,151 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,151 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B 2023-04-06 18:58:38,151 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B 2023-04-06 18:58:38,151 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:38,152 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName B 2023-04-06 18:58:38,152 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:38,152 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,153 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C 2023-04-06 18:58:38,153 DEBUG [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C 2023-04-06 18:58:38,153 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 18:58:38,153 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03155aa75aad25b899de8cf2f21dd327 columnFamilyName C 2023-04-06 18:58:38,153 INFO [StoreOpener-03155aa75aad25b899de8cf2f21dd327-1] regionserver.HStore(310): Store=03155aa75aad25b899de8cf2f21dd327/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 18:58:38,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 18:58:38,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 03155aa75aad25b899de8cf2f21dd327; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63441690, jitterRate=-0.054645150899887085}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 18:58:38,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:38,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., pid=61, masterSystemTime=1680807518139 2023-04-06 18:58:38,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,160 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,160 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807518160"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680807518160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680807518160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680807518160"}]},"ts":"1680807518160"} 2023-04-06 18:58:38,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-06 18:58:38,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 in 174 msec 2023-04-06 18:58:38,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-06 18:58:38,163 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, REOPEN/MOVE in 497 msec 2023-04-06 18:58:38,165 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-06 18:58:38,165 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 503 msec 2023-04-06 18:58:38,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 923 msec 2023-04-06 18:58:38,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-06 18:58:38,168 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec98d63 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,172 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48502121, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,173 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x1ae3ec16 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,176 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8d3d8bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,177 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec0400 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,189 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e6ad437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,189 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x34b8bf14 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,197 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f6f7187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,197 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x387217bb to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,206 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c34fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,206 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x5467be78 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,214 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70ab5f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,214 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x44535519 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,222 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2daeb795, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,223 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7689435c to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,231 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28ac2b09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,231 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x475ee808 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,239 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bf1c01c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,240 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(139): Connect 0x7377b744 to 127.0.0.1:63695 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 18:58:38,247 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43591f8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 18:58:38,257 DEBUG [hconnection-0x6511c27c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,257 DEBUG [hconnection-0x59b02265-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,257 DEBUG [hconnection-0x417b097d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,257 DEBUG [hconnection-0x1320dd6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,257 DEBUG [hconnection-0x63e95888-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,258 DEBUG [hconnection-0x61e6c030-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,258 DEBUG [hconnection-0x1a42c51d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,259 DEBUG [hconnection-0x13ec90d3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,260 DEBUG [hconnection-0x7b532746-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,263 DEBUG [hconnection-0x5e6e485a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,265 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,265 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,266 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,269 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 18:58:38,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,273 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:38,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:58:38,280 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:38,280 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:38,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:38,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:38,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:38,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:38,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,281 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,281 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,281 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,289 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,290 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,290 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,290 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,290 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,290 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,291 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,291 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,297 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3580c734[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,298 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,298 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807578297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807518291, max timestamp=1680807518297Num uniques 859; 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807518291, max timestamp=1680807518297Num uniques 859; 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680807518291, max timestamp=1680807518297Num uniques 859; 2023-04-06 18:58:38,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:38,305 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,305 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,305 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,306 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,306 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,308 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,308 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,308 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,309 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,310 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,311 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,313 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,313 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,313 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,313 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,314 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807578314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807578315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807578315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807578318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807578402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,413 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,416 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,416 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,417 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,417 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,418 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,418 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,418 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,418 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,418 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,419 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807578418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807578418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807578419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,421 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2133a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,421 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,422 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,422 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807578420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,422 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,425 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,425 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,425 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,426 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,426 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,427 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,428 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,429 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:38,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,429 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,429 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,429 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,529 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,532 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,532 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,534 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,534 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,534 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,534 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,534 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,535 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,535 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,535 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,535 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff16e9a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,536 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,537 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,537 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,537 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,537 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,537 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,538 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,538 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,540 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,540 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,540 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,540 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,540 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,540 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,540 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,540 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,541 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,542 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,544 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,544 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,544 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,544 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807578606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807578621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807578622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807578622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807578624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,644 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,647 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,647 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,648 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,648 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,649 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,649 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,650 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,650 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,650 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,650 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,651 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5541206f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,655 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,655 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,655 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,656 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,657 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:38,661 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,661 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406559c8ca5ee444b6989ba3a228d322008 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406559c8ca5ee444b6989ba3a228d322008 2023-04-06 18:58:38,705 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/a7b6d0cf99dc4e0a8a9c73860c6dc637 2023-04-06 18:58:38,761 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,764 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,764 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,765 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,765 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,766 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,767 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,768 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,768 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,768 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,768 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66cbf494[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,769 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,770 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,770 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,772 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,772 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,772 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,773 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,774 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,775 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,779 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,780 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,780 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,880 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,883 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,884 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:38,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:38,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:38,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:38,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:38,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:38,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:38,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:38,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:38,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:38,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:38,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:38,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:38,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:38,886 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:38,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771170b4[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:38,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:38,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,888 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,888 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:38,888 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:38,888 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:38,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:38,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:38,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:38,891 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:38,891 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:38,892 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:38,892 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:38,892 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:38,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:38,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:38,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:38,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:38,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:38,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:38,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:38,896 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:38,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:38,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:38,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,896 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:38,896 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:38,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:38,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807578908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807578925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807578925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807578925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:38,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807578926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:38,996 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:38,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:38,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:38,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:38,999 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:38,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:38,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:38,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:38,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:38,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,000 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,000 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,000 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,000 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:39,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:39,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:39,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:39,002 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,002 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:39,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73905572[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:39,004 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:39,004 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,005 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,005 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:39,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,005 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:39,005 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:39,014 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:39,021 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,021 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,021 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,022 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:39,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:39,023 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:39,023 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:39,023 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:39,023 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:39,024 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:39,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:39,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:39,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:39,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,038 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:39,039 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:39,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/02dd1a4ab90742feb8d756806b887f75 2023-04-06 18:58:39,139 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:39,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:39,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:39,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:39,141 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:39,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:39,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:39,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:39,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:39,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:39,145 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,146 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,146 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:39,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8e5137[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:39,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:39,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:39,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:39,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:39,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:39,148 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:39,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:39,150 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:39,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:39,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:39,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:39,150 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:39,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:39,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,153 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:39,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:39,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,153 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:39,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,253 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:39,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:39,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:39,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:39,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:39,256 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:39,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:39,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:39,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:39,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:39,259 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:39,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:39,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d87f06[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:39,260 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,261 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:39,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:39,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:39,261 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:39,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:39,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:39,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:39,264 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:39,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:39,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:39,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:39,264 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:39,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,265 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,267 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,268 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:39,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:39,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,269 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:39,269 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,369 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:39,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:39,372 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:39,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:39,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:39,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,372 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,373 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,373 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,373 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,373 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:39,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,374 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,374 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,374 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:39,375 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:39,375 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:39,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3029b9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:39,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:39,377 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,377 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:39,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,378 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,378 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,378 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:39,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:39,380 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:39,380 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:39,380 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:39,380 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:39,380 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:39,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,381 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,381 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,382 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:39,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,384 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:39,384 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,384 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:39,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807579421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:39,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807579428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807579430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:39,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807579430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:39,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807579431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,484 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:39,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:39,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:39,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:39,491 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:39,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:39,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:39,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,491 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,492 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,492 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:39,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,494 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,494 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,494 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:39,507 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:39,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:39,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,514 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,514 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:39,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7aca48[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:39,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:39,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:39,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:39,517 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:39,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,517 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:39,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:39,518 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,518 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/41191f20e62540c4b00d8ea6c6e53dd6 2023-04-06 18:58:39,521 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 18:58:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:39,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:39,523 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:39,523 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:39,523 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:39,523 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:39,523 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:39,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/a7b6d0cf99dc4e0a8a9c73860c6dc637 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637 2023-04-06 18:58:39,526 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:39,526 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,527 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:39,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:39,527 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:39,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,529 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637, entries=150, sequenceid=16, filesize=25.1 K 2023-04-06 18:58:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:39,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/02dd1a4ab90742feb8d756806b887f75 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75 2023-04-06 18:58:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:39,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:39,533 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:39,533 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:39,533 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:39,533 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:39,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:39,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 18:58:39,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/41191f20e62540c4b00d8ea6c6e53dd6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6 2023-04-06 18:58:39,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 18:58:39,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 03155aa75aad25b899de8cf2f21dd327 in 1276ms, sequenceid=16, compaction requested=false 2023-04-06 18:58:39,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:39,633 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:39,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:39,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:39,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:39,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:39,636 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:39,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:39,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:39,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,637 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:39,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:39,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:39,637 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:39,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:39,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:39,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:39,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:39,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:39,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:39,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:39,639 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:39,639 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:39,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:39,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:39,639 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-06 18:58:39,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:39,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:39,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:39,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:39,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:39,640 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:40,049 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304067e80777417cd487db37d1e44308dde5b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e80777417cd487db37d1e44308dde5b 2023-04-06 18:58:40,053 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/5a89b0228ad14897b0b37630be377cb5 2023-04-06 18:58:40,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:40,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:40,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807580441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807580441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807580442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807580443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807580443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,459 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/880c779c1f894595aea33231abaff6c8 2023-04-06 18:58:40,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807580547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807580546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807580547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807580550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807580550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807580750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807580750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807580752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807580753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807580754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,867 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/d1a7bac9d45443f2a50a6cdee05557cf 2023-04-06 18:58:40,870 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/5a89b0228ad14897b0b37630be377cb5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5 2023-04-06 18:58:40,873 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5, entries=150, sequenceid=50, filesize=25.1 K 2023-04-06 18:58:40,873 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/880c779c1f894595aea33231abaff6c8 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8 2023-04-06 18:58:40,876 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 18:58:40,876 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/d1a7bac9d45443f2a50a6cdee05557cf as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf 2023-04-06 18:58:40,879 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 18:58:40,886 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 03155aa75aad25b899de8cf2f21dd327 in 1247ms, sequenceid=50, compaction requested=false 2023-04-06 18:58:40,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:40,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:40,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:40,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:40,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:40,886 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:40,887 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:40,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,887 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a037f9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:40,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:40,888 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,889 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:40,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,889 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:40,890 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:40,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:40,891 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:40,891 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:40,891 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:40,891 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:40,891 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:40,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:40,892 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:40,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:40,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:40,893 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,894 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:40,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:40,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,895 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:40,895 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:40,895 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:40,895 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:40,896 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:40,995 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:40,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:40,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:40,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:40,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:40,998 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:40,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:40,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:40,998 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:40,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:40,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:40,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,999 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:40,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:40,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:40,999 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:41,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:41,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:41,000 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:41,000 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:41,000 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:41,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:41,001 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:41,001 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:41,001 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:41,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:41,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:41,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807581064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807521061, max timestamp=1680807521068Num uniques 859; 2023-04-06 18:58:41,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:41,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807521061, max timestamp=1680807521068Num uniques 859; 2023-04-06 18:58:41,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:41,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807521061, max timestamp=1680807521068Num uniques 859; 2023-04-06 18:58:41,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:41,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807581079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807581082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807581083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807581084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807581169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807581184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807581185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807581187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807581189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807581373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807581388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807581390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807581391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807581391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,415 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040622986e3f46f74c6e94abed9ecb817809 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040622986e3f46f74c6e94abed9ecb817809 2023-04-06 18:58:41,419 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/fe8e3fb0123842e5a00186b8be40814a 2023-04-06 18:58:41,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807581676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807581692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807581692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807581693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807581695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,828 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/2ec60978b7a448719918242766ec95c4 2023-04-06 18:58:41,841 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/14bd41a88f784e968aa6c48f786a08c2 2023-04-06 18:58:41,844 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/fe8e3fb0123842e5a00186b8be40814a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a 2023-04-06 18:58:41,847 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a, entries=150, sequenceid=62, filesize=25.1 K 2023-04-06 18:58:41,848 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/2ec60978b7a448719918242766ec95c4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4 2023-04-06 18:58:41,851 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 18:58:41,852 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/14bd41a88f784e968aa6c48f786a08c2 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2 2023-04-06 18:58:41,854 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 18:58:41,855 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 03155aa75aad25b899de8cf2f21dd327 in 854ms, sequenceid=62, compaction requested=true 2023-04-06 18:58:41,855 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:41,855 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:41,855 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:41,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:41,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:41,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d55064[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:41,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:41,857 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,858 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,858 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:41,858 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:41,858 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:41,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,858 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:41,859 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:41,859 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:41,861 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:41,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:41,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:41,861 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:41,861 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:41,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,861 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:41,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,862 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:41,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:41,862 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,863 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:41,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,864 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,864 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,864 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:41,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:41,864 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:41,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:41,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:41,865 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:41,865 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:41,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:41,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:41,965 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:41,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:41,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:41,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:41,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:41,970 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:41,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:41,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:41,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:41,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,970 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,971 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,971 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:41,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:41,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:41,974 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:41,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:41,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:41,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:41,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:41,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:41,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:41,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:41,975 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:41,975 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:41,975 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:42,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:42,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:42,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807582199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807582199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807582199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807582200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807582202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807582303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807582304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807582307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,386 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406501f399576534f9884e54999c9bce3bc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406501f399576534f9884e54999c9bce3bc 2023-04-06 18:58:42,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807582508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807582508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807582512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,790 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/9dd3a616a4a44ac8b08b3121ab61cfe4 2023-04-06 18:58:42,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807582810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807582812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:42,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:42,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807582815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,198 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/8f2d57b933454c5abf769fb58de22371 2023-04-06 18:58:43,205 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/072ab3b16bfa4f4081a3c60df063b6f0 2023-04-06 18:58:43,207 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/9dd3a616a4a44ac8b08b3121ab61cfe4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4 2023-04-06 18:58:43,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,210 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4, entries=150, sequenceid=95, filesize=25.1 K 2023-04-06 18:58:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807583207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,210 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/8f2d57b933454c5abf769fb58de22371 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371 2023-04-06 18:58:43,213 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371, entries=150, sequenceid=95, filesize=11.6 K 2023-04-06 18:58:43,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/072ab3b16bfa4f4081a3c60df063b6f0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0 2023-04-06 18:58:43,215 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0, entries=150, sequenceid=95, filesize=11.6 K 2023-04-06 18:58:43,216 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=60.38 KB/61830 for 03155aa75aad25b899de8cf2f21dd327 in 1241ms, sequenceid=95, compaction requested=true 2023-04-06 18:58:43,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:43,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:43,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:43,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c3ce46[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,222 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,225 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,225 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,226 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,226 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,228 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,229 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,229 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,230 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,230 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,230 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807583274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807583313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807583318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807583320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,330 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,332 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,333 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,335 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,336 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,336 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,336 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,336 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a6639c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,338 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,339 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,339 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,340 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,340 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,341 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,342 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,344 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,344 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,344 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,344 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:43,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807583378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,444 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,447 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,447 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,448 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,449 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,449 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,450 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,451 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72fe852e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,453 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,453 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,455 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,456 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,456 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,457 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,459 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,460 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,460 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,460 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,560 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,562 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,563 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,564 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,565 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,566 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,566 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,566 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,566 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b11ad65[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,568 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,568 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,568 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,568 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,568 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,569 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,569 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,570 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,571 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,572 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,572 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,573 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,575 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,575 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,575 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807583581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,627 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c78b089272da41c482a2a73be283aecb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c78b089272da41c482a2a73be283aecb 2023-04-06 18:58:43,632 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/0d47f398632b4435bee98555b9d06126 2023-04-06 18:58:43,675 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,678 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,678 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,679 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,680 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,680 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,680 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,680 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,681 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,681 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dc6fbbf[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,683 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,684 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,686 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,686 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,686 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,687 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,687 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:43,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,690 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,690 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,690 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,791 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,794 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,795 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,796 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,796 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,797 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b7ad718[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,802 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,803 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,805 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,805 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,805 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,807 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,809 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,809 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,809 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,809 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:43,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807583885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,909 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:43,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:43,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:43,912 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:43,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:43,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:43,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,912 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,913 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,913 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,913 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,913 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:43,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:43,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:43,914 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:43,914 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:43,914 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:43,914 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:43,914 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:43,915 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,915 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d2be0ac[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:43,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:43,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,917 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:43,917 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,917 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:43,917 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:43,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:43,919 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:43,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:43,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:43,919 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:43,919 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:43,920 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,920 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:43,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,920 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:43,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:43,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:43,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:43,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:43,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:43,924 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:43,924 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:43,923 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:43,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:43,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:43,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,024 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:44,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:44,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:44,026 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:44,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:44,026 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:44,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,027 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,028 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,028 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:44,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:44,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:44,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:44,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:44,029 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:44,029 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:44,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,030 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,030 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,030 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:44,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b171ae9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:44,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:44,031 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,032 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:44,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:44,032 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,032 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:44,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:44,033 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:44,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:44,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:44,034 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:44,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:44,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:44,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:44,034 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:44,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,036 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,037 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:44,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:44,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,038 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,039 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:44,039 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:44,039 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/11d641e27c6d490aaac7f4b49a896862 2023-04-06 18:58:44,139 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:44,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:44,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:44,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:44,142 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:44,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:44,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:44,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:44,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:44,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:44,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:44,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:44,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:44,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:44,144 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:44,145 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,145 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:44,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71960f37[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:44,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:44,146 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,146 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:44,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:44,147 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,147 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:44,149 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:44,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:44,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:44,149 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:44,149 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:44,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:44,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,150 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:44,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:44,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,153 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:44,154 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:44,254 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:44,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:44,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:44,256 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:44,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:44,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:44,256 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,257 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,257 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:44,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:44,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:44,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:44,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,258 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:44,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:44,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:44,259 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:44,259 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:44,260 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,260 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c29c40a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:44,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:44,261 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:44,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,262 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,262 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:44,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:44,264 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:44,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:44,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:44,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:44,264 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:44,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,277 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,278 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:44,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,281 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:44,281 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:44,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:44,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807584323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807584325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:44,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807584327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,382 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:44,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:44,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:44,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:44,384 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:44,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:44,384 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:44,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,385 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:44,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:44,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:44,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:44,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:44,386 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,386 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:44,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:44,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:44,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:44,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:44,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:44,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:44,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:44,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e757ba3[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:44,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:44,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:44,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:44,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:44,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:44,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807584390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:44,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:44,392 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:44,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:44,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:44,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:44,393 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:44,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:44,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,393 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:44,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:44,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:44,397 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:44,397 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:44,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/2c761ca9194e40f4a95266ad3a746ef3 2023-04-06 18:58:44,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/0d47f398632b4435bee98555b9d06126 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126 2023-04-06 18:58:44,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126, entries=200, sequenceid=108, filesize=31.9 K 2023-04-06 18:58:44,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/11d641e27c6d490aaac7f4b49a896862 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862 2023-04-06 18:58:44,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862, entries=150, sequenceid=108, filesize=11.6 K 2023-04-06 18:58:44,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/2c761ca9194e40f4a95266ad3a746ef3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3 2023-04-06 18:58:44,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3, entries=150, sequenceid=108, filesize=11.6 K 2023-04-06 18:58:44,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 03155aa75aad25b899de8cf2f21dd327 in 1244ms, sequenceid=108, compaction requested=true 2023-04-06 18:58:44,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:44,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:44,463 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:44,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:44,463 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:44,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:44,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135670 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:44,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:44,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/A is initiating minor compaction (all files) 2023-04-06 18:58:44,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/B is initiating minor compaction (all files) 2023-04-06 18:58:44,467 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/A in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,467 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/B in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,467 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=132.5 K 2023-04-06 18:58:44,467 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=58.0 K 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a7b6d0cf99dc4e0a8a9c73860c6dc637, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680807518276 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 02dd1a4ab90742feb8d756806b887f75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680807518276 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 5a89b0228ad14897b0b37630be377cb5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680807518314 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 880c779c1f894595aea33231abaff6c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680807518314 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting fe8e3fb0123842e5a00186b8be40814a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807520436 2023-04-06 18:58:44,468 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2ec60978b7a448719918242766ec95c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807520436 2023-04-06 18:58:44,469 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 9dd3a616a4a44ac8b08b3121ab61cfe4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680807521077 2023-04-06 18:58:44,469 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8f2d57b933454c5abf769fb58de22371, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680807521077 2023-04-06 18:58:44,469 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0d47f398632b4435bee98555b9d06126, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:44,469 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 11d641e27c6d490aaac7f4b49a896862, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:44,477 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#B#compaction#496 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:44,483 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#A#compaction#495 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:44,497 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:44,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:44,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:44,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:44,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:44,500 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:44,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:44,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:44,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:44,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:44,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:44,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:44,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:44,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:44,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:44,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:44,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:44,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:44,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:44,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:44,503 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,503 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:44,503 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:58:44,503 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:44,504 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:44,504 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:44,504 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:44,504 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:44,504 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:44,887 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/a434690da8e244ebb8816579d303e645 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a434690da8e244ebb8816579d303e645 2023-04-06 18:58:44,890 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/B of 03155aa75aad25b899de8cf2f21dd327 into a434690da8e244ebb8816579d303e645(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:44,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:44,891 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/B, priority=11, startTime=1680807524463; duration=0sec 2023-04-06 18:58:44,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:44,891 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:44,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:44,892 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/C is initiating minor compaction (all files) 2023-04-06 18:58:44,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/C in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:44,892 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=58.0 K 2023-04-06 18:58:44,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 41191f20e62540c4b00d8ea6c6e53dd6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680807518276 2023-04-06 18:58:44,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d1a7bac9d45443f2a50a6cdee05557cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680807518314 2023-04-06 18:58:44,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 14bd41a88f784e968aa6c48f786a08c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680807520436 2023-04-06 18:58:44,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 072ab3b16bfa4f4081a3c60df063b6f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680807521077 2023-04-06 18:58:44,893 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 2c761ca9194e40f4a95266ad3a746ef3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:44,899 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:44,904 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/c759f57d1640469faf74c69e2e968d9a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c759f57d1640469faf74c69e2e968d9a 2023-04-06 18:58:44,907 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/C of 03155aa75aad25b899de8cf2f21dd327 into c759f57d1640469faf74c69e2e968d9a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:44,907 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:44,907 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/C, priority=11, startTime=1680807524463; duration=0sec 2023-04-06 18:58:44,907 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:44,917 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b825678ef6ea477a8865c085d59b8503 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b825678ef6ea477a8865c085d59b8503 2023-04-06 18:58:45,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:45,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:45,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:45,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807585251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:45,320 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=127, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/92d02466ddfa4a55afae838771f61bb6 2023-04-06 18:58:45,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:45,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807585353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:45,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807585397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:45,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:45,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807585557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:45,703 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/1020c643c27e4da28380e21042e2d973 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/1020c643c27e4da28380e21042e2d973 2023-04-06 18:58:45,705 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/A of 03155aa75aad25b899de8cf2f21dd327 into 1020c643c27e4da28380e21042e2d973(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 18:58:45,706 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:45,706 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/A, priority=11, startTime=1680807524463; duration=1sec 2023-04-06 18:58:45,706 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:45,728 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/a7fb769d64614a608f2c9627b0928599 2023-04-06 18:58:45,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:45,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807585862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,136 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/6c3c4eda4f284710b21c810c4909ed17 2023-04-06 18:58:46,139 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/92d02466ddfa4a55afae838771f61bb6 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6 2023-04-06 18:58:46,141 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6, entries=150, sequenceid=127, filesize=25.1 K 2023-04-06 18:58:46,142 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/a7fb769d64614a608f2c9627b0928599 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599 2023-04-06 18:58:46,144 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599, entries=150, sequenceid=127, filesize=11.6 K 2023-04-06 18:58:46,145 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/6c3c4eda4f284710b21c810c4909ed17 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17 2023-04-06 18:58:46,148 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17, entries=150, sequenceid=127, filesize=11.6 K 2023-04-06 18:58:46,148 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 03155aa75aad25b899de8cf2f21dd327 in 1645ms, sequenceid=127, compaction requested=false 2023-04-06 18:58:46,148 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:46,148 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:46,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:46,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:46,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:46,148 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:46,149 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,149 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:46,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:46,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:46,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d66f6d6[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:46,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:46,151 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,152 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:46,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:46,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,152 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:46,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:46,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:46,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:46,153 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,153 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:46,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:46,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:46,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:46,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:46,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:46,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:46,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:46,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,158 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:46,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,159 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:46,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:46,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:46,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:46,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:46,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:46,160 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:46,160 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:46,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:46,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,260 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:46,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:46,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:46,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:46,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:46,262 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:46,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:46,262 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:46,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:46,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,263 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:46,263 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:46,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:46,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:46,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:46,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:46,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:46,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:46,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:46,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:46,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:46,265 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:46,265 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:46,265 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:46,286 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406d87f241415f74ff5aa011be66d4936d4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d87f241415f74ff5aa011be66d4936d4 2023-04-06 18:58:46,295 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/2ebfae576cbe4129be0d6036e837ad66 2023-04-06 18:58:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:46,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807586349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807586352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807586352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807586367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807586453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807586455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807586455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807586655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807586657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807586658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,704 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/c66023bec4e349f4bfe8da3b7b829b97 2023-04-06 18:58:46,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807586960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807586960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:46,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807586963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,118 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/81e075780ab248f39b9e3b37af0a2f0b 2023-04-06 18:58:47,123 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/2ebfae576cbe4129be0d6036e837ad66 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66 2023-04-06 18:58:47,128 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66, entries=150, sequenceid=143, filesize=25.3 K 2023-04-06 18:58:47,129 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/c66023bec4e349f4bfe8da3b7b829b97 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97 2023-04-06 18:58:47,135 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97, entries=150, sequenceid=143, filesize=11.8 K 2023-04-06 18:58:47,136 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/81e075780ab248f39b9e3b37af0a2f0b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b 2023-04-06 18:58:47,138 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b, entries=150, sequenceid=143, filesize=11.8 K 2023-04-06 18:58:47,139 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 03155aa75aad25b899de8cf2f21dd327 in 874ms, sequenceid=143, compaction requested=true 2023-04-06 18:58:47,139 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:47,139 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:47,139 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:47,140 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:47,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:47,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,141 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:47,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:47,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc8abfa[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:47,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:47,142 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,143 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:47,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:47,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,143 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:47,143 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:47,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:47,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:47,144 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,144 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:47,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:47,147 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:47,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:47,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:47,147 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:47,147 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:47,168 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,168 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:47,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,168 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:47,169 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:47,170 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,171 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:47,171 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:47,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:47,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:47,171 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:47,171 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:47,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:47,272 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:47,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:47,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:47,274 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:47,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:47,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:47,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:47,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:47,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:47,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:47,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:47,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:47,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:47,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:47,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:47,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:47,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:47,277 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:47,277 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:47,277 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:47,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:47,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807587403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807587414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,416 DEBUG [Thread-3806] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:58:47,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807587463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807587465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807587466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807587506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:47,692 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406873bacb96fe1446f84c41adfc7db59c4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406873bacb96fe1446f84c41adfc7db59c4 2023-04-06 18:58:47,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807587708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:48,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807588012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,097 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=162, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/0574b2112e54448f91435142777e7c5d 2023-04-06 18:58:48,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:48,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58702 deadline: 1680807588468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58668 deadline: 1680807588468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:48,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58684 deadline: 1680807588473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,505 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/109e4545022e41bfad42b6d97d009402 2023-04-06 18:58:48,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58654 deadline: 1680807588517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,527 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/b2b0478c286e424ca6deef05b10d4bd0 2023-04-06 18:58:48,530 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/0574b2112e54448f91435142777e7c5d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d 2023-04-06 18:58:48,532 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d, entries=150, sequenceid=162, filesize=25.3 K 2023-04-06 18:58:48,533 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/109e4545022e41bfad42b6d97d009402 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402 2023-04-06 18:58:48,536 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402, entries=150, sequenceid=162, filesize=11.8 K 2023-04-06 18:58:48,536 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/b2b0478c286e424ca6deef05b10d4bd0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0 2023-04-06 18:58:48,539 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0, entries=150, sequenceid=162, filesize=11.8 K 2023-04-06 18:58:48,546 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 03155aa75aad25b899de8cf2f21dd327 in 1269ms, sequenceid=162, compaction requested=true 2023-04-06 18:58:48,546 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:48,546 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:48,546 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:48,548 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,548 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,549 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:48,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e033bfd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:48,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:48,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,550 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:48,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:48,550 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:48,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,550 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:48,551 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:48,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:48,551 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:48,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:48,553 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:48,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:48,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:48,554 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:48,554 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:48,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:48,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,554 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:48,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:48,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:48,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:48,557 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:48,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:48,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:48,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:48,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:48,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:48,557 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:48,558 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:48,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,558 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:48,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:48,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:48,658 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:48,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:48,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:48,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:48,660 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:48,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:48,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:48,660 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:48,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:48,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:48,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:48,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:48,662 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:48,662 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:48,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:48,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:48,662 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:48,662 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:48,663 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:48,663 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:48,663 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:48,663 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:48,663 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:48,672 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062563d358682045a0b55f7c8e1e0b0508 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062563d358682045a0b55f7c8e1e0b0508 2023-04-06 18:58:49,076 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=175, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/672938ea0ae346b0b01466e08141db2a 2023-04-06 18:58:49,486 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/8a22303ab78840f89ff25bd27d91b5e3 2023-04-06 18:58:49,493 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/9c50c12499a2453198e8d5e9a98c1ced 2023-04-06 18:58:49,496 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/672938ea0ae346b0b01466e08141db2a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a 2023-04-06 18:58:49,499 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a, entries=150, sequenceid=175, filesize=25.3 K 2023-04-06 18:58:49,500 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/8a22303ab78840f89ff25bd27d91b5e3 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3 2023-04-06 18:58:49,503 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3, entries=150, sequenceid=175, filesize=11.8 K 2023-04-06 18:58:49,503 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/9c50c12499a2453198e8d5e9a98c1ced as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced 2023-04-06 18:58:49,506 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced, entries=150, sequenceid=175, filesize=11.8 K 2023-04-06 18:58:49,506 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 03155aa75aad25b899de8cf2f21dd327 in 843ms, sequenceid=175, compaction requested=true 2023-04-06 18:58:49,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:49,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:49,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:49,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:49,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:49,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:49,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bfee5fd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:49,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:49,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:49,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:49,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:49,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:49,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:49,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:49,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:49,513 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:49,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,513 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:49,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:49,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:49,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,517 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:49,516 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,517 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:49,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,516 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,517 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:49,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:49,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:49,615 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304061eb632c19fbd4a5a931657a132bd11a5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061eb632c19fbd4a5a931657a132bd11a5 2023-04-06 18:58:49,617 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:49,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:49,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:49,634 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:49,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:49,634 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:49,635 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,635 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,635 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,635 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,636 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:49,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:49,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:49,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:49,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,636 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:49,637 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,637 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:49,637 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:49,637 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:49,638 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:49,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61346c12[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:49,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:49,639 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,640 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:49,640 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,640 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:49,640 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:49,640 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,641 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:49,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:49,642 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:49,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:49,642 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:49,643 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:49,643 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:49,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,643 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,644 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,645 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:49,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,646 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,647 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:49,647 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:49,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,646 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,647 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,647 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,647 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,647 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807529646, max timestamp=1680807529666Num uniques 859; 2023-04-06 18:58:49,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:49,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807529646, max timestamp=1680807529666Num uniques 859; 2023-04-06 18:58:49,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:49,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807529646, max timestamp=1680807529666Num uniques 859; 2023-04-06 18:58:49,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:49,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:49,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807589721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,747 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:49,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:49,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:49,749 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:49,749 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,750 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,750 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:49,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:49,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:49,752 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:49,752 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:49,752 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:49,752 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:49,752 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:49,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,753 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,753 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:49,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@488a9cb5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:49,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:49,754 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,756 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:49,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,756 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:49,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:49,756 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:49,757 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:49,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,757 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:49,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:49,759 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:49,759 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:49,759 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:49,760 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,760 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,761 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:49,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,763 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:49,763 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:49,762 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:49,762 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,763 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807589824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,863 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:49,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:49,865 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:49,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,865 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,866 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:49,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:49,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:49,868 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,868 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:49,868 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:49,868 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:49,869 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,869 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,870 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:49,870 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:49,870 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe18a8a[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:49,871 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:49,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,871 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:49,872 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:49,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,872 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:49,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:49,873 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:49,874 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:49,874 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:49,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,874 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,875 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,877 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:49,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:49,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,878 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:49,878 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:49,878 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,978 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:49,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:49,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:49,981 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:49,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:49,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:49,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,982 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,983 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:49,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:49,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:49,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:49,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:49,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:49,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:49,985 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,985 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:49,985 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:49,985 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:49,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:49,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,986 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,986 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:49,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733c5986[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:49,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:49,987 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,988 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:49,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:49,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:49,988 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:49,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:49,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,989 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:49,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:49,991 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:49,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:49,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:49,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:49,991 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:49,991 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,991 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:49,991 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:49,992 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:49,994 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:49,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:49,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:49,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:49,994 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:49,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:49,995 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:49,995 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:49,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:49,995 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:50,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807590028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,036 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/683b919dcc8d4a73b930267009fa4d78 2023-04-06 18:58:50,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/0191d60064ad4049a428a091ea6621a1 2023-04-06 18:58:50,095 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,097 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,098 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,099 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,099 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,100 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,100 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,100 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,100 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312aff0f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,102 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,103 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,103 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,103 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,104 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,104 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,106 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,106 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,109 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,110 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,110 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,111 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,111 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,111 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,211 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,213 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,213 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,214 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,214 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,216 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dd9a2ab[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,219 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,220 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,221 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,221 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,221 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,222 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,224 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,225 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,225 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,225 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,225 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,325 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,328 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,328 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,329 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,330 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,331 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,331 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,331 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1fd840[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,333 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,334 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,334 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:50,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807590333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,335 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,337 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,337 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,337 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,337 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,337 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,339 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,340 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,340 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,341 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,341 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,441 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,444 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,444 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,445 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,446 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,446 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,446 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,446 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,447 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,447 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f7e5a6c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,450 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,450 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/a0199322ee70491d83c0c3f7d308d141 2023-04-06 18:58:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,452 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,452 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,452 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,453 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/683b919dcc8d4a73b930267009fa4d78 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78 2023-04-06 18:58:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,454 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,455 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78, entries=150, sequenceid=185, filesize=25.3 K 2023-04-06 18:58:50,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/0191d60064ad4049a428a091ea6621a1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,458 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,458 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,458 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,458 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,459 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1, entries=100, sequenceid=185, filesize=9.4 K 2023-04-06 18:58:50,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/a0199322ee70491d83c0c3f7d308d141 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141 2023-04-06 18:58:50,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141, entries=100, sequenceid=185, filesize=9.4 K 2023-04-06 18:58:50,464 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 03155aa75aad25b899de8cf2f21dd327 in 865ms, sequenceid=185, compaction requested=true 2023-04-06 18:58:50,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:50,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:50,464 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 18:58:50,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:50,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:50,464 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 18:58:50,465 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155244 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 18:58:50,465 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69694 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 18:58:50,465 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/A is initiating minor compaction (all files) 2023-04-06 18:58:50,465 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/B is initiating minor compaction (all files) 2023-04-06 18:58:50,465 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/A in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,465 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/B in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,465 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/1020c643c27e4da28380e21042e2d973, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=151.6 K 2023-04-06 18:58:50,465 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a434690da8e244ebb8816579d303e645, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=68.1 K 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 1020c643c27e4da28380e21042e2d973, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a434690da8e244ebb8816579d303e645, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 92d02466ddfa4a55afae838771f61bb6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680807523260 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a7fb769d64614a608f2c9627b0928599, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680807523260 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2ebfae576cbe4129be0d6036e837ad66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680807525244 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c66023bec4e349f4bfe8da3b7b829b97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680807525244 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 0574b2112e54448f91435142777e7c5d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680807526341 2023-04-06 18:58:50,466 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 109e4545022e41bfad42b6d97d009402, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680807526341 2023-04-06 18:58:50,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 672938ea0ae346b0b01466e08141db2a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680807527391 2023-04-06 18:58:50,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 8a22303ab78840f89ff25bd27d91b5e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680807527391 2023-04-06 18:58:50,467 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 683b919dcc8d4a73b930267009fa4d78, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807529583 2023-04-06 18:58:50,467 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0191d60064ad4049a428a091ea6621a1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807529583 2023-04-06 18:58:50,473 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#B#compaction#514 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:50,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807529713, max timestamp=1680807530475Num uniques 1018; 2023-04-06 18:58:50,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:50,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807529713, max timestamp=1680807530475Num uniques 1018; 2023-04-06 18:58:50,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:50,478 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#A#compaction#513 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:50,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680807529713, max timestamp=1680807530475Num uniques 1018; 2023-04-06 18:58:50,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 18:58:50,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:50,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807530498, max timestamp=1680807530510Num uniques 859; 2023-04-06 18:58:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807530498, max timestamp=1680807530510Num uniques 859; 2023-04-06 18:58:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:50,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807530498, max timestamp=1680807530510Num uniques 859; 2023-04-06 18:58:50,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:50,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807530529, max timestamp=1680807530538Num uniques 1118; 2023-04-06 18:58:50,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-06 18:58:50,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807530529, max timestamp=1680807530538Num uniques 1118; 2023-04-06 18:58:50,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-06 18:58:50,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680807530529, max timestamp=1680807530538Num uniques 1118; 2023-04-06 18:58:50,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-04-06 18:58:50,559 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,561 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,562 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,562 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,563 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,563 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,564 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,564 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,565 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,565 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ca060b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,566 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,566 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,566 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,566 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,566 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,567 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,566 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,567 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,567 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,567 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,569 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,569 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,569 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,570 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,570 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,570 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,571 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807530563, max timestamp=1680807530571Num uniques 1127; 2023-04-06 18:58:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-06 18:58:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680807530563, max timestamp=1680807530571Num uniques 1127; 2023-04-06 18:58:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,574 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680807530563, max timestamp=1680807530571Num uniques 1127; 2023-04-06 18:58:50,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=6870, suffix heap size=145176,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=150 2023-04-06 18:58:50,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,575 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,575 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,575 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807530591, max timestamp=1680807530599Num uniques 1086; 2023-04-06 18:58:50,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:58:50,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807530591, max timestamp=1680807530599Num uniques 1009; 2023-04-06 18:58:50,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-06 18:58:50,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680807530591, max timestamp=1680807530599Num uniques 1086; 2023-04-06 18:58:50,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 18:58:50,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680807530612, max timestamp=1680807530623Num uniques 1045; 2023-04-06 18:58:50,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 18:58:50,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680807530612, max timestamp=1680807530623Num uniques 1045; 2023-04-06 18:58:50,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 18:58:50,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680807530612, max timestamp=1680807530623Num uniques 1045; 2023-04-06 18:58:50,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 18:58:50,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807530645, max timestamp=1680807530650Num uniques 1104; 2023-04-06 18:58:50,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 18:58:50,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807530645, max timestamp=1680807530650Num uniques 1104; 2023-04-06 18:58:50,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 18:58:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680807530645, max timestamp=1680807530650Num uniques 1104; 2023-04-06 18:58:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 18:58:50,675 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680807530664, max timestamp=1680807530677Num uniques 1063; 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680807530664, max timestamp=1680807530677Num uniques 1063; 2023-04-06 18:58:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680807530664, max timestamp=1680807530677Num uniques 1063; 2023-04-06 18:58:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-06 18:58:50,679 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,681 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,681 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,682 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,682 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,683 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,683 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49663563[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,685 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,685 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,685 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,686 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,686 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,688 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,688 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,688 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,689 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,690 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,691 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,691 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,691 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,691 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,692 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680807530696, max timestamp=1680807530707Num uniques 1122; 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680807530696, max timestamp=1680807530707Num uniques 1122; 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680807530696, max timestamp=1680807530707Num uniques 1122; 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-06 18:58:50,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=12452, suffix heap size=144656,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=272 2023-04-06 18:58:50,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807530724, max timestamp=1680807530737Num uniques 1131; 2023-04-06 18:58:50,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-06 18:58:50,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807530724, max timestamp=1680807530737Num uniques 1131; 2023-04-06 18:58:50,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-06 18:58:50,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680807530724, max timestamp=1680807530737Num uniques 1131; 2023-04-06 18:58:50,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807530744, max timestamp=1680807530761Num uniques 1040; 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807530744, max timestamp=1680807530761Num uniques 1040; 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680807530744, max timestamp=1680807530761Num uniques 1040; 2023-04-06 18:58:50,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 18:58:50,792 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,794 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,794 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,795 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,795 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,796 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,796 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:50,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,797 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,797 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,797 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,797 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,797 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,798 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,798 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699b39d7[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,799 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,800 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,800 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,800 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,802 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,802 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,802 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,803 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,804 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,806 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,806 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,806 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,806 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:50,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807590810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680807530768, max timestamp=1680807530813Num uniques 1149; 2023-04-06 18:58:50,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-06 18:58:50,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680807530768, max timestamp=1680807530813Num uniques 1149; 2023-04-06 18:58:50,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-06 18:58:50,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680807530768, max timestamp=1680807530813Num uniques 1149; 2023-04-06 18:58:50,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680807530830, max timestamp=1680807530848Num uniques 1058; 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680807530830, max timestamp=1680807530848Num uniques 1058; 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680807530830, max timestamp=1680807530848Num uniques 1058; 2023-04-06 18:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-06 18:58:50,881 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/20ac156a1abe40cf911a802737a4000e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/20ac156a1abe40cf911a802737a4000e 2023-04-06 18:58:50,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680807530854, max timestamp=1680807530881Num uniques 1017; 2023-04-06 18:58:50,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-06 18:58:50,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680807530854, max timestamp=1680807530881Num uniques 1017; 2023-04-06 18:58:50,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680807530854, max timestamp=1680807530881Num uniques 1017; 2023-04-06 18:58:50,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-06 18:58:50,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-06 18:58:50,886 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/B of 03155aa75aad25b899de8cf2f21dd327 into 20ac156a1abe40cf911a802737a4000e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:50,886 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:50,886 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/B, priority=10, startTime=1680807530464; duration=0sec 2023-04-06 18:58:50,886 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:50,887 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 18:58:50,888 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69694 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 18:58:50,888 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/C is initiating minor compaction (all files) 2023-04-06 18:58:50,889 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/C in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,889 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c759f57d1640469faf74c69e2e968d9a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=68.1 K 2023-04-06 18:58:50,889 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c759f57d1640469faf74c69e2e968d9a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680807522194 2023-04-06 18:58:50,889 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 6c3c4eda4f284710b21c810c4909ed17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680807523260 2023-04-06 18:58:50,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 81e075780ab248f39b9e3b37af0a2f0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680807525244 2023-04-06 18:58:50,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting b2b0478c286e424ca6deef05b10d4bd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680807526341 2023-04-06 18:58:50,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 9c50c12499a2453198e8d5e9a98c1ced, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1680807527391 2023-04-06 18:58:50,890 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a0199322ee70491d83c0c3f7d308d141, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807529583 2023-04-06 18:58:50,892 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406f035b7a1ed9d438aad5aff7105491eea to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f035b7a1ed9d438aad5aff7105491eea 2023-04-06 18:58:50,892 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/2ac59773f08846de9fba28ace9b3f532 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ac59773f08846de9fba28ace9b3f532 2023-04-06 18:58:50,900 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/A of 03155aa75aad25b899de8cf2f21dd327 into 2ac59773f08846de9fba28ace9b3f532(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:50,900 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:50,900 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/A, priority=10, startTime=1680807530464; duration=0sec 2023-04-06 18:58:50,900 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:50,902 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#C#compaction#516 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:50,906 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:50,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:50,909 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:50,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:50,909 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:50,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,910 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,910 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,911 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:50,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:50,912 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/f18d81e3472b45bb9d309995e59040ac as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f18d81e3472b45bb9d309995e59040ac 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:50,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:50,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:50,912 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:50,912 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,913 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,913 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1680807530906, max timestamp=1680807530913Num uniques 1126; 2023-04-06 18:58:50,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51556, new segment data size=8046, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-06 18:58:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8054, totalHeapSize=18544, min timestamp=1680807530906, max timestamp=1680807530913Num uniques 1126; 2023-04-06 18:58:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680807530906, max timestamp=1680807530913Num uniques 1126; 2023-04-06 18:58:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51564, new segment data size=8054, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-06 18:58:50,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-06 18:58:50,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:50,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@548ba3b2[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:50,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:50,916 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,916 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:50,917 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,917 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:50,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:50,917 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:50,917 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,918 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/C of 03155aa75aad25b899de8cf2f21dd327 into f18d81e3472b45bb9d309995e59040ac(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:50,918 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:50,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:50,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,918 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/C, priority=10, startTime=1680807530464; duration=0sec 2023-04-06 18:58:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,918 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:50,918 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:50,920 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:50,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:50,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:50,920 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:50,920 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:50,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,921 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:50,921 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:50,922 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:50,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:50,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:50,924 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:50,924 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:50,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:50,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:50,925 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:50,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:50,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680807530926, max timestamp=1680807530931Num uniques 1035; 2023-04-06 18:58:50,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-06 18:58:50,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680807530926, max timestamp=1680807530931Num uniques 1035; 2023-04-06 18:58:50,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-06 18:58:50,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680807530926, max timestamp=1680807530931Num uniques 1035; 2023-04-06 18:58:50,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-06 18:58:50,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680807530941, max timestamp=1680807530947Num uniques 1094; 2023-04-06 18:58:50,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 18:58:50,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680807530941, max timestamp=1680807530947Num uniques 1094; 2023-04-06 18:58:50,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 18:58:50,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680807530941, max timestamp=1680807530947Num uniques 1094; 2023-04-06 18:58:50,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680807530958, max timestamp=1680807530964Num uniques 1053; 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680807530958, max timestamp=1680807530964Num uniques 1053; 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680807530958, max timestamp=1680807530964Num uniques 1053; 2023-04-06 18:58:50,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-06 18:58:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680807530979, max timestamp=1680807530985Num uniques 1112; 2023-04-06 18:58:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-06 18:58:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680807530979, max timestamp=1680807530985Num uniques 1112; 2023-04-06 18:58:50,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-06 18:58:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680807530979, max timestamp=1680807530985Num uniques 1112; 2023-04-06 18:58:50,988 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680807531017, max timestamp=1680807531018Num uniques 1021; 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680807531017, max timestamp=1680807531018Num uniques 1021; 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680807531017, max timestamp=1680807531018Num uniques 1021; 2023-04-06 18:58:51,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-06 18:58:51,025 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,027 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,027 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,029 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,029 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,030 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,030 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,031 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,032 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,033 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,034 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d081d76[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,035 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,035 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,035 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,036 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,037 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,038 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,038 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680807531028, max timestamp=1680807531037Num uniques 1030; 2023-04-06 18:58:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12814, totalHeapSize=29360, min timestamp=1680807531028, max timestamp=1680807531037Num uniques 1030; 2023-04-06 18:58:51,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-06 18:58:51,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8230, totalHeapSize=18960, min timestamp=1680807531033, max timestamp=1680807531037Num uniques 1030; 2023-04-06 18:58:51,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47164, new segment data size=12814, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-06 18:58:51,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47160, new segment data size=8230, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-06 18:58:51,048 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,050 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,051 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,051 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,052 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,052 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,052 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,052 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680807531050, max timestamp=1680807531055Num uniques 1139; 2023-04-06 18:58:51,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-06 18:58:51,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680807531050, max timestamp=1680807531055Num uniques 1139; 2023-04-06 18:58:51,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-06 18:58:51,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680807531050, max timestamp=1680807531055Num uniques 1039; 2023-04-06 18:58:51,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1680807531066, max timestamp=1680807531073Num uniques 1098; 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=15938, suffix heap size=142160,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=348 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1680807531066, max timestamp=1680807531073Num uniques 1098; 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=15938, suffix heap size=142160,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=348 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1680807531066, max timestamp=1680807531073Num uniques 1098; 2023-04-06 18:58:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=15938, suffix heap size=142160,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=348 2023-04-06 18:58:51,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807591087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807591088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807591088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807591089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,152 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,155 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,156 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,156 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,157 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,157 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,157 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,157 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,157 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,158 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55bde899[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,160 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,162 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,162 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,162 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,163 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,163 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,165 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,166 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,166 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807591191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807591191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807591191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807591191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,267 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,269 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,269 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,272 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,272 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,273 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,273 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,273 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,273 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,273 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,274 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65582678[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,275 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,275 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,276 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,276 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,276 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,278 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,278 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,278 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,279 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,279 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,280 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,281 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,281 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,282 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,281 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,282 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,298 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/662c9f76326e4edaa7dbc3a6b5b15d6b 2023-04-06 18:58:51,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/0cb11b54b3074147a46e0308e3474bdb 2023-04-06 18:58:51,382 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,385 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,385 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,386 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,386 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,387 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,387 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,388 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,388 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,388 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,389 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c478f52[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,390 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,391 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,391 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,393 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,393 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,394 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,395 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,396 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807591395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,397 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807591395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,397 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,397 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807591396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807591396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:58680 deadline: 1680807591443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,446 DEBUG [Thread-3806] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., hostname=jenkins-hbase20.apache.org,34607,1680807373848, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 18:58:51,497 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,500 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,500 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,501 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,501 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,502 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,502 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,503 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,503 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,504 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149dc01[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,505 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,506 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,508 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,508 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,508 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,510 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,511 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,512 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,512 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,612 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,614 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,614 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,615 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,615 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,616 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,616 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:51,617 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:51,617 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:51,618 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b8a0591[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:51,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:51,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,620 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:51,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:51,621 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:51,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:51,623 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:51,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:51,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:51,623 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:51,623 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:51,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:51,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,626 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:51,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:51,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:51,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,627 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,627 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:51,627 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:51,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807591698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807591698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807591698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807591699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/4715b891a8aa4c17a8485349faefb2bc 2023-04-06 18:58:51,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/662c9f76326e4edaa7dbc3a6b5b15d6b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b 2023-04-06 18:58:51,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b, entries=150, sequenceid=223, filesize=25.3 K 2023-04-06 18:58:51,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/0cb11b54b3074147a46e0308e3474bdb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb 2023-04-06 18:58:51,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb, entries=150, sequenceid=223, filesize=11.8 K 2023-04-06 18:58:51,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/4715b891a8aa4c17a8485349faefb2bc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc 2023-04-06 18:58:51,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc, entries=150, sequenceid=223, filesize=11.8 K 2023-04-06 18:58:51,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=161.02 KB/164880 for 03155aa75aad25b899de8cf2f21dd327 in 1246ms, sequenceid=223, compaction requested=false 2023-04-06 18:58:51,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:51,727 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:51,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:51,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:51,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:51,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:51,729 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:51,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:51,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:51,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,730 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,731 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:51,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:51,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:51,731 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:51,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:51,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:51,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:51,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:51,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:51,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:51,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:51,732 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:51,732 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=161.02 KB heapSize=446.95 KB 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:51,732 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:51,733 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:51,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304064dce1cc69f8740d393d016913d51b421 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064dce1cc69f8740d393d016913d51b421 2023-04-06 18:58:51,771 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=624, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/af7bac4a96204d538423023e094fdfe5 2023-04-06 18:58:52,181 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/a4c2a4bb0c734f579505682260f8210b 2023-04-06 18:58:52,189 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/811f8704faa84b54bb182a164fb3592a 2023-04-06 18:58:52,192 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/af7bac4a96204d538423023e094fdfe5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5 2023-04-06 18:58:52,195 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5, entries=150, sequenceid=624, filesize=25.4 K 2023-04-06 18:58:52,195 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/a4c2a4bb0c734f579505682260f8210b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b 2023-04-06 18:58:52,200 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b, entries=150, sequenceid=624, filesize=11.9 K 2023-04-06 18:58:52,201 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/811f8704faa84b54bb182a164fb3592a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a 2023-04-06 18:58:52,204 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a, entries=150, sequenceid=624, filesize=11.9 K 2023-04-06 18:58:52,204 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~446.91 KB/457632, currentSize=0 B/0 for 03155aa75aad25b899de8cf2f21dd327 in 472ms, sequenceid=624, compaction requested=true 2023-04-06 18:58:52,204 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,204 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,204 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:52,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:52,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:52,205 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:52,205 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,206 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:52,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f458f7e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:52,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:52,207 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:52,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:52,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,218 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:52,218 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:52,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:52,232 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:52,232 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:52,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:52,235 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:52,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:52,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:52,235 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:52,235 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:52,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,243 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,243 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:52,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:52,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:52,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:52,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:52,249 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:52,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:52,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:52,249 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:52,249 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:52,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807532261, max timestamp=1680807532271Num uniques 859; 2023-04-06 18:58:52,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:52,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807532261, max timestamp=1680807532271Num uniques 859; 2023-04-06 18:58:52,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:52,275 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b1dcfffba292411f80222fa4cab67245 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b1dcfffba292411f80222fa4cab67245 2023-04-06 18:58:52,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807532261, max timestamp=1680807532271Num uniques 859; 2023-04-06 18:58:52,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:52,290 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=634, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/6c99c0bb281642479076e351f143ef30 2023-04-06 18:58:52,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807592293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807592293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807592296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/83dadcc656de4e3aa88d2c6a066b9f5a 2023-04-06 18:58:52,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807592297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/553b6aa6e9a048ce9165b6df4b140a28 2023-04-06 18:58:52,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/6c99c0bb281642479076e351f143ef30 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30 2023-04-06 18:58:52,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30, entries=250, sequenceid=634, filesize=39.2 K 2023-04-06 18:58:52,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/83dadcc656de4e3aa88d2c6a066b9f5a as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a 2023-04-06 18:58:52,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a, entries=100, sequenceid=634, filesize=9.5 K 2023-04-06 18:58:52,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/553b6aa6e9a048ce9165b6df4b140a28 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28 2023-04-06 18:58:52,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28, entries=100, sequenceid=634, filesize=9.5 K 2023-04-06 18:58:52,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 03155aa75aad25b899de8cf2f21dd327 in 72ms, sequenceid=634, compaction requested=true 2023-04-06 18:58:52,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:52,320 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:52,320 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:52,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:52,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:52,321 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118340 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:52,321 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46376 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:52,321 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/A is initiating minor compaction (all files) 2023-04-06 18:58:52,321 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/B is initiating minor compaction (all files) 2023-04-06 18:58:52,321 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/A in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,321 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/B in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,321 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ac59773f08846de9fba28ace9b3f532, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=115.6 K 2023-04-06 18:58:52,321 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/20ac156a1abe40cf911a802737a4000e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=45.3 K 2023-04-06 18:58:52,321 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 2ac59773f08846de9fba28ace9b3f532, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807527399 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 20ac156a1abe40cf911a802737a4000e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807527399 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 662c9f76326e4edaa7dbc3a6b5b15d6b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1680807529717 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 0cb11b54b3074147a46e0308e3474bdb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1680807529717 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting af7bac4a96204d538423023e094fdfe5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1680807531085 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting a4c2a4bb0c734f579505682260f8210b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1680807531085 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 6c99c0bb281642479076e351f143ef30, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807532228 2023-04-06 18:58:52,322 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 83dadcc656de4e3aa88d2c6a066b9f5a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807532231 2023-04-06 18:58:52,328 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#B#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:52,331 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#A#compaction#526 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:52,349 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:52,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:52,351 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/a0d658809f374b008c9170bf00032b7d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a0d658809f374b008c9170bf00032b7d 2023-04-06 18:58:52,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:52,352 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:52,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:52,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:52,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:52,352 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,353 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:52,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:52,353 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,354 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/A of 03155aa75aad25b899de8cf2f21dd327 into a0d658809f374b008c9170bf00032b7d(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:52,354 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,354 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/A, priority=12, startTime=1680807532320; duration=0sec 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,354 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:52,354 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:52,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:52,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:52,354 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:52,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:52,355 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:52,355 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46376 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 18:58:52,355 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-06 18:58:52,355 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/C is initiating minor compaction (all files) 2023-04-06 18:58:52,355 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/C in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:52,355 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f18d81e3472b45bb9d309995e59040ac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=45.3 K 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:52,355 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f18d81e3472b45bb9d309995e59040ac, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680807527399 2023-04-06 18:58:52,355 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:52,356 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4715b891a8aa4c17a8485349faefb2bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1680807529717 2023-04-06 18:58:52,356 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 811f8704faa84b54bb182a164fb3592a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1680807531085 2023-04-06 18:58:52,356 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 553b6aa6e9a048ce9165b6df4b140a28, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807532231 2023-04-06 18:58:52,362 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#C#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:52,367 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040681f047cf15604da081a28c6c8e303eba to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681f047cf15604da081a28c6c8e303eba 2023-04-06 18:58:52,369 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/3eb03b7e7528440d8bf43e2817ef31ee as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/3eb03b7e7528440d8bf43e2817ef31ee 2023-04-06 18:58:52,372 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=671, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/4a9b09bf409640fe8cbd489d61f71593 2023-04-06 18:58:52,374 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/C of 03155aa75aad25b899de8cf2f21dd327 into 3eb03b7e7528440d8bf43e2817ef31ee(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:52,374 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,374 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/C, priority=12, startTime=1680807532320; duration=0sec 2023-04-06 18:58:52,374 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:52,386 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/d29a8d9bc455424daa9d1685c5415a42 2023-04-06 18:58:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:52,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807592403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807592403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807592404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807592405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807592507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807592507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807592508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807592509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807592710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807592710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807592710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:52,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807592711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,739 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/d2e96cdb85834563a47fbeace9665992 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d2e96cdb85834563a47fbeace9665992 2023-04-06 18:58:52,742 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/B of 03155aa75aad25b899de8cf2f21dd327 into d2e96cdb85834563a47fbeace9665992(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:52,742 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,742 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/B, priority=12, startTime=1680807532320; duration=0sec 2023-04-06 18:58:52,742 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:52,795 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/4bc271b91ffc49deaa299420b9341212 2023-04-06 18:58:52,798 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/4a9b09bf409640fe8cbd489d61f71593 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593 2023-04-06 18:58:52,801 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593, entries=150, sequenceid=671, filesize=25.4 K 2023-04-06 18:58:52,802 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/d29a8d9bc455424daa9d1685c5415a42 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42 2023-04-06 18:58:52,805 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42, entries=150, sequenceid=671, filesize=11.9 K 2023-04-06 18:58:52,806 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/4bc271b91ffc49deaa299420b9341212 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212 2023-04-06 18:58:52,809 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212, entries=150, sequenceid=671, filesize=11.9 K 2023-04-06 18:58:52,809 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 03155aa75aad25b899de8cf2f21dd327 in 454ms, sequenceid=671, compaction requested=false 2023-04-06 18:58:52,809 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:52,809 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:52,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:52,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,809 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:52,810 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,810 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,811 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:52,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa8bedb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:52,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:52,812 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,812 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:52,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,813 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:52,813 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,813 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:52,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:52,815 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:52,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:52,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:52,815 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:52,815 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:52,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:52,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,816 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:52,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:52,817 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,818 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:52,820 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:52,820 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:52,820 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:52,820 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:52,920 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:52,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:52,923 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:52,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:52,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:52,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:52,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,923 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,924 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:52,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:52,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:52,924 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:52,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:52,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:52,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:52,925 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:52,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:52,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:52,925 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 18:58:52,925 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:52,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:52,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:52,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:52,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:52,926 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:53,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807533045, max timestamp=1680807533051Num uniques 859; 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807533045, max timestamp=1680807533051Num uniques 859; 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680807533045, max timestamp=1680807533051Num uniques 859; 2023-04-06 18:58:53,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 18:58:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807593065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807593068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807593068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807593069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807593170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807593172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807593172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807593172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,338 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069478e2b62f744b8e89d52e23395892fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069478e2b62f744b8e89d52e23395892fd 2023-04-06 18:58:53,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807593373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807593374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807593374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807593375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807593675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807593678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807593679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807593679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,742 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=682, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/204127dcdd2940c4b98f553042725555 2023-04-06 18:58:53,749 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/5dfc51c44fd24232af45dc2147fb7983 2023-04-06 18:58:53,756 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/237ad39231d54bba997689eb98f72c8c 2023-04-06 18:58:53,758 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/204127dcdd2940c4b98f553042725555 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555 2023-04-06 18:58:53,763 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555, entries=100, sequenceid=682, filesize=18.6 K 2023-04-06 18:58:53,764 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/5dfc51c44fd24232af45dc2147fb7983 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983 2023-04-06 18:58:53,766 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983, entries=100, sequenceid=682, filesize=9.5 K 2023-04-06 18:58:53,767 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/237ad39231d54bba997689eb98f72c8c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c 2023-04-06 18:58:53,769 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c, entries=100, sequenceid=682, filesize=9.5 K 2023-04-06 18:58:53,770 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 03155aa75aad25b899de8cf2f21dd327 in 845ms, sequenceid=682, compaction requested=true 2023-04-06 18:58:53,770 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:53,770 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:53,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:53,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:53,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:53,770 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:53,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,771 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:53,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,771 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:53,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ecafb[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:53,772 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:53,772 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,773 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,773 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:53,773 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:53,773 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:53,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,773 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:53,774 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:53,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:53,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:53,775 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:53,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:53,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:53,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:53,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:53,776 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:53,776 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:53,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,777 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:53,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:53,777 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:53,778 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:53,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:53,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:53,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:53,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:53,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:53,779 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:53,780 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:53,780 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:53,779 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:53,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:53,780 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,880 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:53,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:53,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:53,882 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:53,882 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:53,882 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:53,882 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:53,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:53,883 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:53,883 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:53,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:53,884 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:53,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:53,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:53,884 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:53,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:53,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:53,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:53,885 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:53,885 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:53,885 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:53,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:53,886 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:53,894 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304063da52bbf286547afbeb359c634906085 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063da52bbf286547afbeb359c634906085 2023-04-06 18:58:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:54,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:54,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807594185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807594185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807594186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807594187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807594289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807594289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807594289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,299 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=718, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/f7ca3045c877454f88049f91e13fe080 2023-04-06 18:58:54,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807594491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807594492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807594492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,706 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/4f9dd01184fb4ba2b07466a088ef1a52 2023-04-06 18:58:54,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807594796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807594796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:54,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807594797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,114 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/c49b6673f83149b285be6cd8f77db6fc 2023-04-06 18:58:55,117 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/f7ca3045c877454f88049f91e13fe080 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080 2023-04-06 18:58:55,120 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080, entries=150, sequenceid=718, filesize=25.4 K 2023-04-06 18:58:55,120 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/4f9dd01184fb4ba2b07466a088ef1a52 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52 2023-04-06 18:58:55,123 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52, entries=150, sequenceid=718, filesize=11.9 K 2023-04-06 18:58:55,123 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/c49b6673f83149b285be6cd8f77db6fc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc 2023-04-06 18:58:55,126 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc, entries=150, sequenceid=718, filesize=11.9 K 2023-04-06 18:58:55,126 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 03155aa75aad25b899de8cf2f21dd327 in 1241ms, sequenceid=718, compaction requested=true 2023-04-06 18:58:55,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:55,126 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,126 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,127 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,127 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,128 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b83e1ee[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,129 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,129 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,129 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,129 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,130 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,129 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,130 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,130 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,132 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,132 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,132 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,132 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,132 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,133 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,134 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,135 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,136 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,136 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,136 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,136 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,137 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:55,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:55,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:55,207 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406cb4844e8caa1442488be05fba4eb9a8f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb4844e8caa1442488be05fba4eb9a8f 2023-04-06 18:58:55,210 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=728, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/4ef1e9bf8fa44771823dc6f7296c9c19 2023-04-06 18:58:55,237 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,239 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,239 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,240 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,240 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,241 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,241 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,241 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,241 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,242 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,243 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@580ead9f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,244 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,245 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,245 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,245 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,246 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,247 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,247 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,247 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,248 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,248 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,248 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,249 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,250 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,251 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,251 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,251 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,251 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807535236, max timestamp=1680807535255Num uniques 859; 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807535236, max timestamp=1680807535255Num uniques 859; 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680807535236, max timestamp=1680807535255Num uniques 859; 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:55,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 18:58:55,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807595304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807595304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807595307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807595308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,351 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,354 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,354 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,355 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,355 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,356 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,357 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,357 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,357 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,357 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,357 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,358 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,358 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525a8e23[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,359 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,360 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,360 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,360 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,360 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,360 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,360 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,361 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,363 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,363 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,363 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,364 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,364 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,365 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,367 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,367 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,368 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,368 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,368 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807595408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807595408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807595411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807595411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,468 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,471 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,471 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,472 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,472 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,473 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,473 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,473 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,474 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,474 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,474 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,474 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,475 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,475 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24359f81[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,476 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,476 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,476 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,477 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,477 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,477 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,479 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,479 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,479 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,480 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,480 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,481 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,483 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,483 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,483 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,483 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,583 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,586 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,586 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,587 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,587 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,588 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,589 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,588 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,589 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,589 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,589 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,590 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50cd9678[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,591 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,591 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,591 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,592 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,592 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,594 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,594 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,594 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,595 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,595 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,597 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,598 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,598 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,599 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,599 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807595611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807595611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807595614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807595615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/4d08eb3805c6443ab10387a436dba348 2023-04-06 18:58:55,699 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,701 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,701 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,702 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,702 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,703 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,703 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,703 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,704 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,704 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,705 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,705 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d02bbab[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,706 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,706 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,707 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,707 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,707 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,714 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,717 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,717 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,717 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,723 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,724 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,725 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,727 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:55,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,727 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,728 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,728 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,728 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,828 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,830 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,831 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,832 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,832 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,833 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,833 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,834 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,834 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,835 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,835 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2f09be[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,836 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,837 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,837 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,838 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,840 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,840 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,840 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,841 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,841 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,841 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,842 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,843 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,845 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,846 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,846 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,846 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,846 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807595914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807595915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807595917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807595919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,946 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:55,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:55,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:55,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:55,949 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:55,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:55,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:55,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,949 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,950 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,950 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,951 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:55,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:55,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:55,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:55,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:55,952 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:55,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:55,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:55,953 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,954 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:55,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ee58cd[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:55,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:55,956 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,956 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:55,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:55,956 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:55,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,957 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:55,957 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:55,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,957 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:55,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:55,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:55,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:55,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:55,959 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:55,959 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:55,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,960 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,961 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,962 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:55,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:55,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:55,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:55,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:55,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:55,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:55,963 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:55,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:55,963 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:55,964 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:55,964 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:55,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:55,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:55,964 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:55,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/f16f98f26f964a9681f0e03dddc26c70 2023-04-06 18:58:56,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/4ef1e9bf8fa44771823dc6f7296c9c19 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19 2023-04-06 18:58:56,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19, entries=200, sequenceid=728, filesize=32.3 K 2023-04-06 18:58:56,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/4d08eb3805c6443ab10387a436dba348 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348 2023-04-06 18:58:56,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348, entries=150, sequenceid=728, filesize=11.9 K 2023-04-06 18:58:56,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/f16f98f26f964a9681f0e03dddc26c70 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70 2023-04-06 18:58:56,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70, entries=150, sequenceid=728, filesize=11.9 K 2023-04-06 18:58:56,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 03155aa75aad25b899de8cf2f21dd327 in 844ms, sequenceid=728, compaction requested=true 2023-04-06 18:58:56,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,042 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:56,042 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:56,042 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:56,042 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:56,042 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:56,043 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130734 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:56,044 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:56,044 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/A is initiating minor compaction (all files) 2023-04-06 18:58:56,044 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/B is initiating minor compaction (all files) 2023-04-06 18:58:56,044 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/A in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,044 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/B in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,044 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a0d658809f374b008c9170bf00032b7d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=127.7 K 2023-04-06 18:58:56,044 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d2e96cdb85834563a47fbeace9665992, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=57.6 K 2023-04-06 18:58:56,044 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting a0d658809f374b008c9170bf00032b7d, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807531087 2023-04-06 18:58:56,044 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d2e96cdb85834563a47fbeace9665992, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807531087 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4a9b09bf409640fe8cbd489d61f71593, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1680807532290 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting d29a8d9bc455424daa9d1685c5415a42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1680807532290 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 204127dcdd2940c4b98f553042725555, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1680807532403 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 5dfc51c44fd24232af45dc2147fb7983, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1680807532403 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting f7ca3045c877454f88049f91e13fe080, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680807533067 2023-04-06 18:58:56,045 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4f9dd01184fb4ba2b07466a088ef1a52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680807533067 2023-04-06 18:58:56,046 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 4ef1e9bf8fa44771823dc6f7296c9c19, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:56,046 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4d08eb3805c6443ab10387a436dba348, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:56,053 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#B#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:56,058 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#A#compaction#540 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:56,060 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/929d702663464291a5fa8fb4ab0b84c7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/929d702663464291a5fa8fb4ab0b84c7 2023-04-06 18:58:56,064 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,064 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/B of 03155aa75aad25b899de8cf2f21dd327 into 929d702663464291a5fa8fb4ab0b84c7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:56,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,064 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/B, priority=11, startTime=1680807536042; duration=0sec 2023-04-06 18:58:56,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:56,064 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 18:58:56,066 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 18:58:56,066 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/C is initiating minor compaction (all files) 2023-04-06 18:58:56,066 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/C in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,066 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/3eb03b7e7528440d8bf43e2817ef31ee, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=57.6 K 2023-04-06 18:58:56,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,066 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 3eb03b7e7528440d8bf43e2817ef31ee, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680807531087 2023-04-06 18:58:56,067 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 4bc271b91ffc49deaa299420b9341212, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1680807532290 2023-04-06 18:58:56,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,071 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 237ad39231d54bba997689eb98f72c8c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1680807532403 2023-04-06 18:58:56,071 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting c49b6673f83149b285be6cd8f77db6fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680807533067 2023-04-06 18:58:56,072 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting f16f98f26f964a9681f0e03dddc26c70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:56,083 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#C#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,086 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,086 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,087 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,087 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,088 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,090 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,090 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,091 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:56,091 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:56,094 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/524afcec04214e18a3c8b93062cad3c5 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/524afcec04214e18a3c8b93062cad3c5 2023-04-06 18:58:56,098 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/C of 03155aa75aad25b899de8cf2f21dd327 into 524afcec04214e18a3c8b93062cad3c5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:56,098 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,098 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/C, priority=11, startTime=1680807536042; duration=0sec 2023-04-06 18:58:56,098 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:56,109 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065666e271b02743c5a7b47bc3eca55e25 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065666e271b02743c5a7b47bc3eca55e25 2023-04-06 18:58:56,113 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/d0f0510f7e4c4d1ca36d5035982ff87b 2023-04-06 18:58:56,127 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/7f9e2528accd446a83de3834144f0d90 2023-04-06 18:58:56,137 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/547d57e65029405aaf7a497e3a1e426e 2023-04-06 18:58:56,140 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/d0f0510f7e4c4d1ca36d5035982ff87b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b 2023-04-06 18:58:56,142 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b, entries=150, sequenceid=766, filesize=25.4 K 2023-04-06 18:58:56,143 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/7f9e2528accd446a83de3834144f0d90 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90 2023-04-06 18:58:56,146 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90, entries=150, sequenceid=766, filesize=11.9 K 2023-04-06 18:58:56,146 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/547d57e65029405aaf7a497e3a1e426e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e 2023-04-06 18:58:56,149 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e, entries=150, sequenceid=766, filesize=11.9 K 2023-04-06 18:58:56,150 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 03155aa75aad25b899de8cf2f21dd327 in 59ms, sequenceid=766, compaction requested=false 2023-04-06 18:58:56,150 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,150 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,150 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,151 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,151 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,152 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f2c7e9e[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,153 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,154 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,155 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,155 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,157 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,157 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,157 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,158 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,159 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,160 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,161 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,161 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,162 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,161 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,262 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,264 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,264 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,265 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,266 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,266 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,267 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,267 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,267 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,268 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,268 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,268 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,269 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,269 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@279b5b33[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,270 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,271 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,271 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,271 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,272 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,272 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,274 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,274 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,274 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,275 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,282 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,283 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,285 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,285 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,285 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,285 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,286 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,286 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,286 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,386 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,388 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,388 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,389 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,389 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,390 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,390 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,391 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,391 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,391 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,391 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,391 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,392 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,392 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,393 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24fad487[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,394 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,394 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,394 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,395 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,395 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,397 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,397 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,397 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,398 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,398 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,399 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,400 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,401 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,401 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,401 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,401 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:56,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 18:58:56,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:56,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:56,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:56,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:56,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:56,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807536457, max timestamp=1680807536460Num uniques 859; 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807536457, max timestamp=1680807536460Num uniques 859; 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680807536457, max timestamp=1680807536460Num uniques 859; 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:56,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 18:58:56,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807596478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807596478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807596479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807596480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,501 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,504 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,504 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,505 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,505 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:56,506 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,506 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,507 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25d3d8e5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,509 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,509 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,509 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,510 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,512 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,512 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,512 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,513 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,514 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,515 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,516 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,516 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807596582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807596583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807596584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807596584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,616 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,618 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,618 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,619 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,619 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,620 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:56,621 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,621 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,622 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,623 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a79a4f[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,624 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,624 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,624 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,625 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,625 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,625 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,627 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,627 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,627 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,628 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,628 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,629 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,631 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,631 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,631 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,631 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,731 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,734 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,734 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,735 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,735 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,736 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,736 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,737 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:56,737 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,737 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,737 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,738 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74981a8b[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,739 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,739 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,740 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,740 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,740 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,742 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,742 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,742 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,743 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,745 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,746 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,746 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,746 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,746 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,746 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,749 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807596785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807596785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807596788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:56,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807596788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,846 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,849 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,849 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,850 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,850 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,851 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,852 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,852 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,852 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:56,852 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,852 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,853 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,854 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b4a673[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,855 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040650b46a27a606409f88897f0ca8873159 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040650b46a27a606409f88897f0ca8873159 2023-04-06 18:58:56,855 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,855 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,856 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,856 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,856 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,857 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,857 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,860 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,860 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,860 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,861 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=779, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/e107171ba7cf4a5a95ffe22d7286a6e1 2023-04-06 18:58:56,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,865 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,866 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,867 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,876 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,876 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,876 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,876 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,876 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,876 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,877 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,877 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,895 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/36409412ac0643bda1e2262804218ab4 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/36409412ac0643bda1e2262804218ab4 2023-04-06 18:58:56,898 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/A of 03155aa75aad25b899de8cf2f21dd327 into 36409412ac0643bda1e2262804218ab4(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:56,898 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:56,899 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/A, priority=11, startTime=1680807536042; duration=0sec 2023-04-06 18:58:56,899 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:56,977 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:56,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:56,979 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:56,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:56,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:56,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,980 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,981 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,981 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:56,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:56,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:56,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:56,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:56,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:56,982 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:56,982 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:56,983 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,983 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,984 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d9b42b0[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,985 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:56,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:56,985 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,985 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:56,987 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,987 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:56,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:56,988 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:56,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:56,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:56,988 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:56,989 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:56,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,989 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,990 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:56,991 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:56,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:56,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:56,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:56,993 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:56,993 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:56,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:56,993 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:56,993 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807597088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807597089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807597091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807597091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,093 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,096 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,096 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,097 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,097 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,098 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,098 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,099 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,099 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,100 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8dbd2ae[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,101 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,101 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,101 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,102 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,102 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,104 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,104 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,104 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,105 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,105 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,106 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,107 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,108 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,108 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,108 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,108 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,208 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,210 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,211 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,212 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,212 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,213 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,213 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,214 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,214 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c23dd7c[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,215 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,216 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,216 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,216 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,217 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,217 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,219 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,219 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,219 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,220 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,220 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,222 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,223 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,223 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,223 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,224 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,224 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,224 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,224 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/908989ae8376489691ffb0a701c5eea7 2023-04-06 18:58:57,324 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,326 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,326 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,327 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,327 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,328 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,328 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,328 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,328 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,329 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,329 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,329 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,330 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0773b5[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,331 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,331 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,331 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,332 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,332 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,332 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,334 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,334 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,334 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,335 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,336 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,336 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,338 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,338 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,338 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,338 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,438 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,441 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,441 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,442 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,442 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,443 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,443 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,444 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,444 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786e3c27[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,445 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,446 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,446 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,446 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,446 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,448 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,448 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,448 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,449 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,450 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,451 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,451 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,452 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,452 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,452 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,552 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,555 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,555 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,556 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,556 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,557 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,557 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,557 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,557 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,557 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,558 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,559 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef6cf26[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,560 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,560 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,560 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,561 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,561 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,561 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,563 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,563 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,563 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,564 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,564 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,564 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,565 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,566 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,567 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,567 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,568 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,567 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,568 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58702 deadline: 1680807597592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58668 deadline: 1680807597594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34607] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58654 deadline: 1680807597596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 18:58:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34607] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:58684 deadline: 1680807597598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,668 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,670 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,670 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,671 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,671 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,672 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,672 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,672 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:57,672 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:57,672 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:57,673 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,673 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:57,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9c3fb9[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:57,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:57,674 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,675 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:57,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:57,675 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,675 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:57,675 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:57,676 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,676 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:57,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:57,677 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:57,677 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:57,677 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:57,677 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:57,677 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:57,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=779 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/64326b4dafd347a6a6e7fe32cf90af5b 2023-04-06 18:58:57,678 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,678 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:57,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,678 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,679 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,680 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:57,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/e107171ba7cf4a5a95ffe22d7286a6e1 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1 2023-04-06 18:58:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1, entries=350, sequenceid=779, filesize=52.9 K 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/908989ae8376489691ffb0a701c5eea7 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7 2023-04-06 18:58:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,684 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:57,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 18:58:57,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:57,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,684 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:57,684 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:57,684 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,686 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7, entries=150, sequenceid=779, filesize=11.9 K 2023-04-06 18:58:57,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/64326b4dafd347a6a6e7fe32cf90af5b as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b 2023-04-06 18:58:57,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b, entries=150, sequenceid=779, filesize=11.9 K 2023-04-06 18:58:57,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 03155aa75aad25b899de8cf2f21dd327 in 1245ms, sequenceid=779, compaction requested=true 2023-04-06 18:58:57,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:57,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:57,690 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:57,690 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:57,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:57,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:57,691 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:57,691 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:57,691 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/A is initiating minor compaction (all files) 2023-04-06 18:58:57,691 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/B is initiating minor compaction (all files) 2023-04-06 18:58:57,691 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/A in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,691 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/B in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,691 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/36409412ac0643bda1e2262804218ab4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=104.5 K 2023-04-06 18:58:57,691 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/929d702663464291a5fa8fb4ab0b84c7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=36.4 K 2023-04-06 18:58:57,692 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 36409412ac0643bda1e2262804218ab4, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:57,692 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 929d702663464291a5fa8fb4ab0b84c7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:57,692 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting d0f0510f7e4c4d1ca36d5035982ff87b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680807535286 2023-04-06 18:58:57,692 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting e107171ba7cf4a5a95ffe22d7286a6e1, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1680807536430 2023-04-06 18:58:57,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 7f9e2528accd446a83de3834144f0d90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680807535286 2023-04-06 18:58:57,693 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] compactions.Compactor(207): Compacting 908989ae8376489691ffb0a701c5eea7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1680807536430 2023-04-06 18:58:57,698 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#B#compaction#550 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:57,699 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#A#compaction#549 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:57,718 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/4f5b7627c74e41c1842b643904db382e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4f5b7627c74e41c1842b643904db382e 2023-04-06 18:58:57,722 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/A of 03155aa75aad25b899de8cf2f21dd327 into 4f5b7627c74e41c1842b643904db382e(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:57,722 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:57,722 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/A, priority=13, startTime=1680807537690; duration=0sec 2023-04-06 18:58:57,722 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 18:58:57,723 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 18:58:57,724 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 18:58:57,724 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1912): 03155aa75aad25b899de8cf2f21dd327/C is initiating minor compaction (all files) 2023-04-06 18:58:57,724 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 03155aa75aad25b899de8cf2f21dd327/C in TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,724 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/524afcec04214e18a3c8b93062cad3c5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b] into tmpdir=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp, totalSize=36.4 K 2023-04-06 18:58:57,724 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 524afcec04214e18a3c8b93062cad3c5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=728, earliestPutTs=1680807534182 2023-04-06 18:58:57,724 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 547d57e65029405aaf7a497e3a1e426e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680807535286 2023-04-06 18:58:57,725 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] compactions.Compactor(207): Compacting 64326b4dafd347a6a6e7fe32cf90af5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=779, earliestPutTs=1680807536430 2023-04-06 18:58:57,730 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] throttle.PressureAwareThroughputController(145): 03155aa75aad25b899de8cf2f21dd327#C#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 18:58:57,736 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/4a1228631a5b4f84a8d979bcf235774e as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4a1228631a5b4f84a8d979bcf235774e 2023-04-06 18:58:57,738 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/C of 03155aa75aad25b899de8cf2f21dd327 into 4a1228631a5b4f84a8d979bcf235774e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:57,738 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:57,738 INFO [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/C, priority=13, startTime=1680807537690; duration=0sec 2023-04-06 18:58:57,738 DEBUG [RS:0;jenkins-hbase20:34607-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:57,784 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:58:57,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 18:58:57,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 18:58:57,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 18:58:57,787 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 18:58:57,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 18:58:57,787 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 18:58:57,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,788 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,788 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:57,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:57,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:57,789 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 18:58:57,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 18:58:57,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 18:58:57,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 18:58:57,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 18:58:57,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 18:58:57,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 18:58:57,789 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 18:58:57,789 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. started... 2023-04-06 18:58:57,790 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:58:57,790 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 18:58:57,804 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b8589e396e224b8fb17727d5c66a52fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8589e396e224b8fb17727d5c66a52fd 2023-04-06 18:58:57,808 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=819, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/d63cdd77bbf6413a9e04e25f80d47521 2023-04-06 18:58:58,109 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/2c7a1d7f5eac406fb77332588ee68874 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2c7a1d7f5eac406fb77332588ee68874 2023-04-06 18:58:58,113 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 03155aa75aad25b899de8cf2f21dd327/B of 03155aa75aad25b899de8cf2f21dd327 into 2c7a1d7f5eac406fb77332588ee68874(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 18:58:58,113 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:58,113 INFO [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327., storeName=03155aa75aad25b899de8cf2f21dd327/B, priority=13, startTime=1680807537690; duration=0sec 2023-04-06 18:58:58,113 DEBUG [RS:0;jenkins-hbase20:34607-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 18:58:58,215 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/ce4f2d70f3624250827ec08526f2fb94 2023-04-06 18:58:58,252 DEBUG [Thread-3815] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7689435c to 127.0.0.1:63695 2023-04-06 18:58:58,252 DEBUG [Thread-3815] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,253 DEBUG [Thread-3811] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5467be78 to 127.0.0.1:63695 2023-04-06 18:58:58,253 DEBUG [Thread-3811] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,254 DEBUG [Thread-3817] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x475ee808 to 127.0.0.1:63695 2023-04-06 18:58:58,254 DEBUG [Thread-3817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,255 DEBUG [Thread-3819] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7377b744 to 127.0.0.1:63695 2023-04-06 18:58:58,255 DEBUG [Thread-3819] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,255 DEBUG [Thread-3813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44535519 to 127.0.0.1:63695 2023-04-06 18:58:58,255 DEBUG [Thread-3813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34607] regionserver.HRegion(9158): Flush requested on 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:58:58,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. as already flushing 2023-04-06 18:58:58,598 DEBUG [Thread-3808] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x387217bb to 127.0.0.1:63695 2023-04-06 18:58:58,598 DEBUG [Thread-3808] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,605 DEBUG [Thread-3804] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec0400 to 127.0.0.1:63695 2023-04-06 18:58:58,605 DEBUG [Thread-3804] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,606 DEBUG [Thread-3802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ae3ec16 to 127.0.0.1:63695 2023-04-06 18:58:58,606 DEBUG [Thread-3802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,607 DEBUG [Thread-3800] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec98d63 to 127.0.0.1:63695 2023-04-06 18:58:58,607 DEBUG [Thread-3800] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:58:58,626 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/7e16fb71cbc747f4aefe20e28b21a621 2023-04-06 18:58:58,633 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/d63cdd77bbf6413a9e04e25f80d47521 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d63cdd77bbf6413a9e04e25f80d47521 2023-04-06 18:58:58,638 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d63cdd77bbf6413a9e04e25f80d47521, entries=150, sequenceid=819, filesize=25.4 K 2023-04-06 18:58:58,639 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/ce4f2d70f3624250827ec08526f2fb94 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/ce4f2d70f3624250827ec08526f2fb94 2023-04-06 18:58:58,644 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/ce4f2d70f3624250827ec08526f2fb94, entries=150, sequenceid=819, filesize=11.9 K 2023-04-06 18:58:58,645 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/7e16fb71cbc747f4aefe20e28b21a621 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/7e16fb71cbc747f4aefe20e28b21a621 2023-04-06 18:58:58,648 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/7e16fb71cbc747f4aefe20e28b21a621, entries=150, sequenceid=819, filesize=11.9 K 2023-04-06 18:58:58,649 INFO [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=26.84 KB/27480 for 03155aa75aad25b899de8cf2f21dd327 in 859ms, sequenceid=819, compaction requested=false 2023-04-06 18:58:58,649 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:58:58,649 DEBUG [rs(jenkins-hbase20.apache.org,34607,1680807373848)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:58:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 18:58:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 18:58:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 18:58:58,649 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 18:58:58,650 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,650 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,651 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 18:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34607,1680807373848' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 18:58:58,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28226f44[Count = 0] remaining members to acquire global barrier 2023-04-06 18:58:58,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 18:58:58,652 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,653 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 18:58:58,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,653 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 18:58:58,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 18:58:58,653 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34607,1680807373848' in zk 2023-04-06 18:58:58,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 18:58:58,654 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:58,654 DEBUG [member: 'jenkins-hbase20.apache.org,34607,1680807373848' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 18:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34607,1680807373848': 2023-04-06 18:58:58,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34607,1680807373848' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 18:58:58,656 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 18:58:58,656 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 18:58:58,656 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 18:58:58,656 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 18:58:58,656 INFO [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 18:58:58,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 18:58:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 18:58:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,657 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 18:58:58,658 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,659 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 18:58:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 18:58:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 18:58:58,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:58,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 18:58:58,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 18:58:58,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 18:58:58,661 DEBUG [(jenkins-hbase20.apache.org,45203,1680807372735)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 18:58:58,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 18:58:58,662 DEBUG [Thread-3810] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 18:58:58,662 DEBUG [Thread-3810] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 18:58:58,661 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 18:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 18:58:58,662 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:58:58,662 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,662 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 18:58:58,662 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 18:58:58,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 18:58:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 18:58:58,762 DEBUG [Thread-3810] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 18:58:58,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 18:59:01,543 DEBUG [Thread-3806] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b8bf14 to 127.0.0.1:63695 2023-04-06 18:59:01,543 DEBUG [Thread-3806] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:01,543 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 18:59:01,544 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 175 2023-04-06 18:59:01,544 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-06 18:59:01,544 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 197 2023-04-06 18:59:01,544 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(394): wrote 178 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 5293 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 5207 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 5287 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 5324 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(398): read 5208 2023-04-06 18:59:01,545 INFO [Listener at localhost.localdomain/41219] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 18:59:01,546 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:59:01,546 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x630ea023 to 127.0.0.1:63695 2023-04-06 18:59:01,546 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:01,547 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 18:59:01,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 18:59:01,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:01,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:01,552 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807541552"}]},"ts":"1680807541552"} 2023-04-06 18:59:01,553 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 18:59:01,555 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 18:59:01,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, UNASSIGN}] 2023-04-06 18:59:01,557 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, UNASSIGN 2023-04-06 18:59:01,558 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:59:01,558 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807541558"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680807541558"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680807541558"}]},"ts":"1680807541558"} 2023-04-06 18:59:01,560 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848}] 2023-04-06 18:59:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:01,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:59:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 03155aa75aad25b899de8cf2f21dd327, disabling compactions & flushes 2023-04-06 18:59:01,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:59:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:59:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. after waiting 0 ms 2023-04-06 18:59:01,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:59:01,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 03155aa75aad25b899de8cf2f21dd327 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=A 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=B 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 03155aa75aad25b899de8cf2f21dd327, store=C 2023-04-06 18:59:01,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 18:59:01,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040677292629047c426290e1d24aba20556c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040677292629047c426290e1d24aba20556c 2023-04-06 18:59:01,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:02,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=828, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/499789ea0b524be18e4e86d7a4733c4f 2023-04-06 18:59:02,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:02,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/eaaabaa2631442d3ac688391b2b05192 2023-04-06 18:59:02,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:02,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/0bfc551cbee649899890c7df33c92871 2023-04-06 18:59:02,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/A/499789ea0b524be18e4e86d7a4733c4f as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/499789ea0b524be18e4e86d7a4733c4f 2023-04-06 18:59:02,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/499789ea0b524be18e4e86d7a4733c4f, entries=100, sequenceid=828, filesize=18.6 K 2023-04-06 18:59:02,975 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/B/eaaabaa2631442d3ac688391b2b05192 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/eaaabaa2631442d3ac688391b2b05192 2023-04-06 18:59:02,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/eaaabaa2631442d3ac688391b2b05192, entries=100, sequenceid=828, filesize=9.5 K 2023-04-06 18:59:02,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/.tmp/C/0bfc551cbee649899890c7df33c92871 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/0bfc551cbee649899890c7df33c92871 2023-04-06 18:59:02,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/0bfc551cbee649899890c7df33c92871, entries=100, sequenceid=828, filesize=9.5 K 2023-04-06 18:59:02,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 03155aa75aad25b899de8cf2f21dd327 in 1274ms, sequenceid=828, compaction requested=true 2023-04-06 18:59:03,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/1020c643c27e4da28380e21042e2d973, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ac59773f08846de9fba28ace9b3f532, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a0d658809f374b008c9170bf00032b7d, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/36409412ac0643bda1e2262804218ab4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1] to archive 2023-04-06 18:59:03,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:59:03,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a7b6d0cf99dc4e0a8a9c73860c6dc637 2023-04-06 18:59:03,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/5a89b0228ad14897b0b37630be377cb5 2023-04-06 18:59:03,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/fe8e3fb0123842e5a00186b8be40814a 2023-04-06 18:59:03,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/9dd3a616a4a44ac8b08b3121ab61cfe4 2023-04-06 18:59:03,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0d47f398632b4435bee98555b9d06126 2023-04-06 18:59:03,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/1020c643c27e4da28380e21042e2d973 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/1020c643c27e4da28380e21042e2d973 2023-04-06 18:59:03,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/92d02466ddfa4a55afae838771f61bb6 2023-04-06 18:59:03,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ebfae576cbe4129be0d6036e837ad66 2023-04-06 18:59:03,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/0574b2112e54448f91435142777e7c5d 2023-04-06 18:59:03,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/672938ea0ae346b0b01466e08141db2a 2023-04-06 18:59:03,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ac59773f08846de9fba28ace9b3f532 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/2ac59773f08846de9fba28ace9b3f532 2023-04-06 18:59:03,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/683b919dcc8d4a73b930267009fa4d78 2023-04-06 18:59:03,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/662c9f76326e4edaa7dbc3a6b5b15d6b 2023-04-06 18:59:03,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/af7bac4a96204d538423023e094fdfe5 2023-04-06 18:59:03,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/6c99c0bb281642479076e351f143ef30 2023-04-06 18:59:03,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a0d658809f374b008c9170bf00032b7d to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/a0d658809f374b008c9170bf00032b7d 2023-04-06 18:59:03,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4a9b09bf409640fe8cbd489d61f71593 2023-04-06 18:59:03,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/204127dcdd2940c4b98f553042725555 2023-04-06 18:59:03,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/f7ca3045c877454f88049f91e13fe080 2023-04-06 18:59:03,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4ef1e9bf8fa44771823dc6f7296c9c19 2023-04-06 18:59:03,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/36409412ac0643bda1e2262804218ab4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/36409412ac0643bda1e2262804218ab4 2023-04-06 18:59:03,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d0f0510f7e4c4d1ca36d5035982ff87b 2023-04-06 18:59:03,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/e107171ba7cf4a5a95ffe22d7286a6e1 2023-04-06 18:59:03,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a434690da8e244ebb8816579d303e645, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/20ac156a1abe40cf911a802737a4000e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d2e96cdb85834563a47fbeace9665992, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/929d702663464291a5fa8fb4ab0b84c7, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7] to archive 2023-04-06 18:59:03,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:59:03,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/02dd1a4ab90742feb8d756806b887f75 2023-04-06 18:59:03,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/880c779c1f894595aea33231abaff6c8 2023-04-06 18:59:03,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2ec60978b7a448719918242766ec95c4 2023-04-06 18:59:03,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8f2d57b933454c5abf769fb58de22371 2023-04-06 18:59:03,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a434690da8e244ebb8816579d303e645 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a434690da8e244ebb8816579d303e645 2023-04-06 18:59:03,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/11d641e27c6d490aaac7f4b49a896862 2023-04-06 18:59:03,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a7fb769d64614a608f2c9627b0928599 2023-04-06 18:59:03,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/c66023bec4e349f4bfe8da3b7b829b97 2023-04-06 18:59:03,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/109e4545022e41bfad42b6d97d009402 2023-04-06 18:59:03,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/8a22303ab78840f89ff25bd27d91b5e3 2023-04-06 18:59:03,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/20ac156a1abe40cf911a802737a4000e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/20ac156a1abe40cf911a802737a4000e 2023-04-06 18:59:03,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0191d60064ad4049a428a091ea6621a1 2023-04-06 18:59:03,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/0cb11b54b3074147a46e0308e3474bdb 2023-04-06 18:59:03,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/a4c2a4bb0c734f579505682260f8210b 2023-04-06 18:59:03,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d2e96cdb85834563a47fbeace9665992 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d2e96cdb85834563a47fbeace9665992 2023-04-06 18:59:03,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/83dadcc656de4e3aa88d2c6a066b9f5a 2023-04-06 18:59:03,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/d29a8d9bc455424daa9d1685c5415a42 2023-04-06 18:59:03,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/5dfc51c44fd24232af45dc2147fb7983 2023-04-06 18:59:03,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4f9dd01184fb4ba2b07466a088ef1a52 2023-04-06 18:59:03,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/929d702663464291a5fa8fb4ab0b84c7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/929d702663464291a5fa8fb4ab0b84c7 2023-04-06 18:59:03,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/4d08eb3805c6443ab10387a436dba348 2023-04-06 18:59:03,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/7f9e2528accd446a83de3834144f0d90 2023-04-06 18:59:03,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/908989ae8376489691ffb0a701c5eea7 2023-04-06 18:59:03,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c759f57d1640469faf74c69e2e968d9a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f18d81e3472b45bb9d309995e59040ac, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/3eb03b7e7528440d8bf43e2817ef31ee, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/524afcec04214e18a3c8b93062cad3c5, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b] to archive 2023-04-06 18:59:03,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 18:59:03,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/41191f20e62540c4b00d8ea6c6e53dd6 2023-04-06 18:59:03,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/d1a7bac9d45443f2a50a6cdee05557cf 2023-04-06 18:59:03,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/14bd41a88f784e968aa6c48f786a08c2 2023-04-06 18:59:03,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/072ab3b16bfa4f4081a3c60df063b6f0 2023-04-06 18:59:03,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c759f57d1640469faf74c69e2e968d9a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c759f57d1640469faf74c69e2e968d9a 2023-04-06 18:59:03,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/2c761ca9194e40f4a95266ad3a746ef3 2023-04-06 18:59:03,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/6c3c4eda4f284710b21c810c4909ed17 2023-04-06 18:59:03,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/81e075780ab248f39b9e3b37af0a2f0b 2023-04-06 18:59:03,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/b2b0478c286e424ca6deef05b10d4bd0 2023-04-06 18:59:03,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/9c50c12499a2453198e8d5e9a98c1ced 2023-04-06 18:59:03,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f18d81e3472b45bb9d309995e59040ac to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f18d81e3472b45bb9d309995e59040ac 2023-04-06 18:59:03,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/a0199322ee70491d83c0c3f7d308d141 2023-04-06 18:59:03,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4715b891a8aa4c17a8485349faefb2bc 2023-04-06 18:59:03,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/811f8704faa84b54bb182a164fb3592a 2023-04-06 18:59:03,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/3eb03b7e7528440d8bf43e2817ef31ee to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/3eb03b7e7528440d8bf43e2817ef31ee 2023-04-06 18:59:03,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/553b6aa6e9a048ce9165b6df4b140a28 2023-04-06 18:59:03,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4bc271b91ffc49deaa299420b9341212 2023-04-06 18:59:03,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/237ad39231d54bba997689eb98f72c8c 2023-04-06 18:59:03,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/c49b6673f83149b285be6cd8f77db6fc 2023-04-06 18:59:03,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/524afcec04214e18a3c8b93062cad3c5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/524afcec04214e18a3c8b93062cad3c5 2023-04-06 18:59:03,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/f16f98f26f964a9681f0e03dddc26c70 2023-04-06 18:59:03,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/547d57e65029405aaf7a497e3a1e426e 2023-04-06 18:59:03,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/64326b4dafd347a6a6e7fe32cf90af5b 2023-04-06 18:59:03,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits/831.seqid, newMaxSeqId=831, maxSeqId=4 2023-04-06 18:59:03,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327. 2023-04-06 18:59:03,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 03155aa75aad25b899de8cf2f21dd327: 2023-04-06 18:59:03,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:59:03,090 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=03155aa75aad25b899de8cf2f21dd327, regionState=CLOSED 2023-04-06 18:59:03,090 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680807543090"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680807543090"}]},"ts":"1680807543090"} 2023-04-06 18:59:03,093 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-06 18:59:03,093 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 03155aa75aad25b899de8cf2f21dd327, server=jenkins-hbase20.apache.org,34607,1680807373848 in 1.5330 sec 2023-04-06 18:59:03,095 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-06 18:59:03,095 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=03155aa75aad25b899de8cf2f21dd327, UNASSIGN in 1.5370 sec 2023-04-06 18:59:03,096 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680807543096"}]},"ts":"1680807543096"} 2023-04-06 18:59:03,097 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 18:59:03,098 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 18:59:03,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5510 sec 2023-04-06 18:59:03,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 18:59:03,660 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-06 18:59:03,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 18:59:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,664 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,665 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 18:59:03,672 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:59:03,677 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C, FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits] 2023-04-06 18:59:03,683 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/499789ea0b524be18e4e86d7a4733c4f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/499789ea0b524be18e4e86d7a4733c4f 2023-04-06 18:59:03,686 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4f5b7627c74e41c1842b643904db382e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/4f5b7627c74e41c1842b643904db382e 2023-04-06 18:59:03,689 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d63cdd77bbf6413a9e04e25f80d47521 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/A/d63cdd77bbf6413a9e04e25f80d47521 2023-04-06 18:59:03,693 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2c7a1d7f5eac406fb77332588ee68874 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/2c7a1d7f5eac406fb77332588ee68874 2023-04-06 18:59:03,695 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/ce4f2d70f3624250827ec08526f2fb94 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/ce4f2d70f3624250827ec08526f2fb94 2023-04-06 18:59:03,697 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/eaaabaa2631442d3ac688391b2b05192 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/B/eaaabaa2631442d3ac688391b2b05192 2023-04-06 18:59:03,701 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/0bfc551cbee649899890c7df33c92871 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/0bfc551cbee649899890c7df33c92871 2023-04-06 18:59:03,703 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4a1228631a5b4f84a8d979bcf235774e to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/4a1228631a5b4f84a8d979bcf235774e 2023-04-06 18:59:03,704 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/7e16fb71cbc747f4aefe20e28b21a621 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/C/7e16fb71cbc747f4aefe20e28b21a621 2023-04-06 18:59:03,708 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits/831.seqid to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327/recovered.edits/831.seqid 2023-04-06 18:59:03,709 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/.tmp/data/default/TestAcidGuarantees/03155aa75aad25b899de8cf2f21dd327 2023-04-06 18:59:03,709 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 18:59:03,709 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:59:03,710 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 18:59:03,715 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061eb632c19fbd4a5a931657a132bd11a5 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061eb632c19fbd4a5a931657a132bd11a5 2023-04-06 18:59:03,717 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040622986e3f46f74c6e94abed9ecb817809 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040622986e3f46f74c6e94abed9ecb817809 2023-04-06 18:59:03,719 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062563d358682045a0b55f7c8e1e0b0508 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062563d358682045a0b55f7c8e1e0b0508 2023-04-06 18:59:03,721 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063da52bbf286547afbeb359c634906085 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063da52bbf286547afbeb359c634906085 2023-04-06 18:59:03,722 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064dce1cc69f8740d393d016913d51b421 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064dce1cc69f8740d393d016913d51b421 2023-04-06 18:59:03,723 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406501f399576534f9884e54999c9bce3bc to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406501f399576534f9884e54999c9bce3bc 2023-04-06 18:59:03,724 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040650b46a27a606409f88897f0ca8873159 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040650b46a27a606409f88897f0ca8873159 2023-04-06 18:59:03,725 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406559c8ca5ee444b6989ba3a228d322008 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406559c8ca5ee444b6989ba3a228d322008 2023-04-06 18:59:03,726 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065666e271b02743c5a7b47bc3eca55e25 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065666e271b02743c5a7b47bc3eca55e25 2023-04-06 18:59:03,727 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040677292629047c426290e1d24aba20556c to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040677292629047c426290e1d24aba20556c 2023-04-06 18:59:03,728 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e80777417cd487db37d1e44308dde5b to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067e80777417cd487db37d1e44308dde5b 2023-04-06 18:59:03,729 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681f047cf15604da081a28c6c8e303eba to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040681f047cf15604da081a28c6c8e303eba 2023-04-06 18:59:03,730 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406873bacb96fe1446f84c41adfc7db59c4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406873bacb96fe1446f84c41adfc7db59c4 2023-04-06 18:59:03,731 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069478e2b62f744b8e89d52e23395892fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069478e2b62f744b8e89d52e23395892fd 2023-04-06 18:59:03,732 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b1dcfffba292411f80222fa4cab67245 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b1dcfffba292411f80222fa4cab67245 2023-04-06 18:59:03,733 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b825678ef6ea477a8865c085d59b8503 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b825678ef6ea477a8865c085d59b8503 2023-04-06 18:59:03,733 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8589e396e224b8fb17727d5c66a52fd to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b8589e396e224b8fb17727d5c66a52fd 2023-04-06 18:59:03,734 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c78b089272da41c482a2a73be283aecb to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c78b089272da41c482a2a73be283aecb 2023-04-06 18:59:03,735 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb4844e8caa1442488be05fba4eb9a8f to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cb4844e8caa1442488be05fba4eb9a8f 2023-04-06 18:59:03,736 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d87f241415f74ff5aa011be66d4936d4 to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406d87f241415f74ff5aa011be66d4936d4 2023-04-06 18:59:03,737 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f035b7a1ed9d438aad5aff7105491eea to hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f035b7a1ed9d438aad5aff7105491eea 2023-04-06 18:59:03,737 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 18:59:03,739 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,740 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 18:59:03,741 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 18:59:03,742 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,742 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 18:59:03,742 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680807543742"}]},"ts":"9223372036854775807"} 2023-04-06 18:59:03,743 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 18:59:03,743 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 03155aa75aad25b899de8cf2f21dd327, NAME => 'TestAcidGuarantees,,1680807515111.03155aa75aad25b899de8cf2f21dd327.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 18:59:03,743 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 18:59:03,743 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680807543743"}]},"ts":"9223372036854775807"} 2023-04-06 18:59:03,744 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 18:59:03,745 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 18:59:03,746 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 84 msec 2023-04-06 18:59:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45203] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 18:59:03,767 INFO [Listener at localhost.localdomain/41219] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-06 18:59:03,780 INFO [Listener at localhost.localdomain/41219] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227 (was 228), OpenFileDescriptor=577 (was 527) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=378 (was 437), ProcessCount=166 (was 166), AvailableMemoryMB=3857 (was 3807) - AvailableMemoryMB LEAK? - 2023-04-06 18:59:03,781 INFO [Listener at localhost.localdomain/41219] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-06 18:59:03,781 INFO [Listener at localhost.localdomain/41219] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 18:59:03,781 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1545112e to 127.0.0.1:63695 2023-04-06 18:59:03,781 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:03,781 DEBUG [Listener at localhost.localdomain/41219] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-06 18:59:03,781 DEBUG [Listener at localhost.localdomain/41219] util.JVMClusterUtil(257): Found active master hash=1701932655, stopped=false 2023-04-06 18:59:03,781 INFO [Listener at localhost.localdomain/41219] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:59:03,782 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 18:59:03,782 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 18:59:03,782 INFO [Listener at localhost.localdomain/41219] procedure2.ProcedureExecutor(629): Stopping 2023-04-06 18:59:03,782 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:59:03,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 18:59:03,782 DEBUG [Listener at localhost.localdomain/41219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38302e48 to 127.0.0.1:63695 2023-04-06 18:59:03,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 18:59:03,783 DEBUG [Listener at localhost.localdomain/41219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:03,783 INFO [Listener at localhost.localdomain/41219] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34607,1680807373848' ***** 2023-04-06 18:59:03,783 INFO [Listener at localhost.localdomain/41219] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-06 18:59:03,783 INFO [RS:0;jenkins-hbase20:34607] regionserver.HeapMemoryManager(220): Stopping 2023-04-06 18:59:03,783 INFO [RS:0;jenkins-hbase20:34607] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-06 18:59:03,783 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-06 18:59:03,783 INFO [RS:0;jenkins-hbase20:34607] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-06 18:59:03,783 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(3304): Received CLOSE for c1504e0a89a4202f82012694b63b21af 2023-04-06 18:59:03,784 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:59:03,784 DEBUG [RS:0;jenkins-hbase20:34607] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x261697dd to 127.0.0.1:63695 2023-04-06 18:59:03,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c1504e0a89a4202f82012694b63b21af, disabling compactions & flushes 2023-04-06 18:59:03,785 DEBUG [RS:0;jenkins-hbase20:34607] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:03,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:59:03,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:59:03,785 INFO [RS:0;jenkins-hbase20:34607] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-06 18:59:03,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. after waiting 0 ms 2023-04-06 18:59:03,785 INFO [RS:0;jenkins-hbase20:34607] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-06 18:59:03,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:59:03,785 INFO [RS:0;jenkins-hbase20:34607] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-06 18:59:03,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c1504e0a89a4202f82012694b63b21af 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-06 18:59:03,785 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-06 18:59:03,785 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-06 18:59:03,786 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 18:59:03,786 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, c1504e0a89a4202f82012694b63b21af=hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af.} 2023-04-06 18:59:03,786 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 18:59:03,786 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 18:59:03,786 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 18:59:03,786 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 18:59:03,786 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-06 18:59:03,787 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740, c1504e0a89a4202f82012694b63b21af 2023-04-06 18:59:03,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/.tmp/info/e66579c729a948bbb300d88037d83d8d 2023-04-06 18:59:03,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/.tmp/info/e66579c729a948bbb300d88037d83d8d as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/info/e66579c729a948bbb300d88037d83d8d 2023-04-06 18:59:03,802 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/info/e66579c729a948bbb300d88037d83d8d, entries=2, sequenceid=6, filesize=4.8 K 2023-04-06 18:59:03,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c1504e0a89a4202f82012694b63b21af in 18ms, sequenceid=6, compaction requested=false 2023-04-06 18:59:03,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/namespace/c1504e0a89a4202f82012694b63b21af/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-06 18:59:03,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:59:03,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c1504e0a89a4202f82012694b63b21af: 2023-04-06 18:59:03,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680807376031.c1504e0a89a4202f82012694b63b21af. 2023-04-06 18:59:03,874 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-06 18:59:03,988 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:04,188 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:04,195 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/info/24a4e975873c4865a9ebb5880947b0bc 2023-04-06 18:59:04,202 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 24a4e975873c4865a9ebb5880947b0bc 2023-04-06 18:59:04,389 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:04,589 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:04,610 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/rep_barrier/9035a43bac8948888f4de82ca2f3c32c 2023-04-06 18:59:04,617 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9035a43bac8948888f4de82ca2f3c32c 2023-04-06 18:59:04,631 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-06 18:59:04,631 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-06 18:59:04,789 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-06 18:59:04,789 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-06 18:59:04,790 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:04,990 DEBUG [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-06 18:59:05,026 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/table/1600d7ff050648949453932f5d332efb 2023-04-06 18:59:05,032 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1600d7ff050648949453932f5d332efb 2023-04-06 18:59:05,033 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/info/24a4e975873c4865a9ebb5880947b0bc as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/info/24a4e975873c4865a9ebb5880947b0bc 2023-04-06 18:59:05,040 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 24a4e975873c4865a9ebb5880947b0bc 2023-04-06 18:59:05,040 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/info/24a4e975873c4865a9ebb5880947b0bc, entries=22, sequenceid=93, filesize=7.1 K 2023-04-06 18:59:05,042 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/rep_barrier/9035a43bac8948888f4de82ca2f3c32c as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/rep_barrier/9035a43bac8948888f4de82ca2f3c32c 2023-04-06 18:59:05,048 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9035a43bac8948888f4de82ca2f3c32c 2023-04-06 18:59:05,048 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/rep_barrier/9035a43bac8948888f4de82ca2f3c32c, entries=6, sequenceid=93, filesize=5.5 K 2023-04-06 18:59:05,049 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/.tmp/table/1600d7ff050648949453932f5d332efb as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/table/1600d7ff050648949453932f5d332efb 2023-04-06 18:59:05,055 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1600d7ff050648949453932f5d332efb 2023-04-06 18:59:05,055 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/table/1600d7ff050648949453932f5d332efb, entries=9, sequenceid=93, filesize=5.5 K 2023-04-06 18:59:05,056 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1270ms, sequenceid=93, compaction requested=false 2023-04-06 18:59:05,065 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-06 18:59:05,065 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 18:59:05,066 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 18:59:05,066 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 18:59:05,066 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-06 18:59:05,190 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34607,1680807373848; all regions closed. 2023-04-06 18:59:05,201 DEBUG [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/oldWALs 2023-04-06 18:59:05,201 INFO [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34607%2C1680807373848.meta:.meta(num 1680807375782) 2023-04-06 18:59:05,208 DEBUG [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/oldWALs 2023-04-06 18:59:05,208 INFO [RS:0;jenkins-hbase20:34607] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34607%2C1680807373848:(num 1680807375692) 2023-04-06 18:59:05,208 DEBUG [RS:0;jenkins-hbase20:34607] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:05,208 INFO [RS:0;jenkins-hbase20:34607] regionserver.LeaseManager(133): Closed leases 2023-04-06 18:59:05,209 INFO [RS:0;jenkins-hbase20:34607] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-06 18:59:05,209 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 18:59:05,210 INFO [RS:0;jenkins-hbase20:34607] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34607 2023-04-06 18:59:05,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 18:59:05,218 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34607,1680807373848 2023-04-06 18:59:05,219 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@575c039c rejected from java.util.concurrent.ThreadPoolExecutor@53638daf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2748] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,219 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 18:59:05,220 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@10cab8aa rejected from java.util.concurrent.ThreadPoolExecutor@53638daf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2748] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,220 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34607,1680807373848] 2023-04-06 18:59:05,221 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34607,1680807373848; numProcessing=1 2023-04-06 18:59:05,222 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34607,1680807373848 already deleted, retry=false 2023-04-06 18:59:05,222 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34607,1680807373848 expired; onlineServers=0 2023-04-06 18:59:05,222 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,45203,1680807372735' ***** 2023-04-06 18:59:05,222 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-06 18:59:05,222 DEBUG [M:0;jenkins-hbase20:45203] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@244c6454, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 18:59:05,222 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,45203,1680807372735 2023-04-06 18:59:05,222 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,45203,1680807372735; all regions closed. 2023-04-06 18:59:05,222 DEBUG [M:0;jenkins-hbase20:45203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 18:59:05,222 DEBUG [M:0;jenkins-hbase20:45203] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-06 18:59:05,222 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-06 18:59:05,222 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680807375321] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680807375321,5,FailOnTimeoutGroup] 2023-04-06 18:59:05,222 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680807375322] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680807375322,5,FailOnTimeoutGroup] 2023-04-06 18:59:05,222 DEBUG [M:0;jenkins-hbase20:45203] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-06 18:59:05,222 INFO [M:0;jenkins-hbase20:45203] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-06 18:59:05,222 INFO [M:0;jenkins-hbase20:45203] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-06 18:59:05,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-06 18:59:05,223 INFO [M:0;jenkins-hbase20:45203] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-06 18:59:05,223 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 18:59:05,223 DEBUG [M:0;jenkins-hbase20:45203] master.HMaster(1512): Stopping service threads 2023-04-06 18:59:05,223 INFO [M:0;jenkins-hbase20:45203] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-06 18:59:05,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 18:59:05,224 ERROR [M:0;jenkins-hbase20:45203] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1561327437) connection to localhost.localdomain/127.0.0.1:34815 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-06 18:59:05,224 INFO [M:0;jenkins-hbase20:45203] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-06 18:59:05,224 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-06 18:59:05,224 DEBUG [M:0;jenkins-hbase20:45203] zookeeper.ZKUtil(398): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-06 18:59:05,224 WARN [M:0;jenkins-hbase20:45203] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-06 18:59:05,224 INFO [M:0;jenkins-hbase20:45203] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-06 18:59:05,225 INFO [M:0;jenkins-hbase20:45203] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-06 18:59:05,225 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 18:59:05,225 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:59:05,225 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:59:05,225 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 18:59:05,225 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:59:05,225 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.11 KB heapSize=393.77 KB 2023-04-06 18:59:05,233 INFO [M:0;jenkins-hbase20:45203] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.11 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/dc1ce3330a7d483fb9f90a2017fd2bf0 2023-04-06 18:59:05,237 INFO [M:0;jenkins-hbase20:45203] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dc1ce3330a7d483fb9f90a2017fd2bf0 2023-04-06 18:59:05,237 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/dc1ce3330a7d483fb9f90a2017fd2bf0 as hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/dc1ce3330a7d483fb9f90a2017fd2bf0 2023-04-06 18:59:05,240 INFO [M:0;jenkins-hbase20:45203] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for dc1ce3330a7d483fb9f90a2017fd2bf0 2023-04-06 18:59:05,240 INFO [M:0;jenkins-hbase20:45203] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34815/user/jenkins/test-data/bc560437-131d-fb50-3ecd-cc40cc45d356/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/dc1ce3330a7d483fb9f90a2017fd2bf0, entries=65, sequenceid=578, filesize=24.7 K 2023-04-06 18:59:05,240 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegion(2948): Finished flush of dataSize ~345.11 KB/353388, heapSize ~393.76 KB/403208, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 15ms, sequenceid=578, compaction requested=false 2023-04-06 18:59:05,242 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 18:59:05,242 DEBUG [M:0;jenkins-hbase20:45203] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 18:59:05,244 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 18:59:05,244 INFO [M:0;jenkins-hbase20:45203] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-06 18:59:05,245 INFO [M:0;jenkins-hbase20:45203] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:45203 2023-04-06 18:59:05,246 DEBUG [M:0;jenkins-hbase20:45203] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,45203,1680807372735 already deleted, retry=false 2023-04-06 18:59:05,322 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 18:59:05,322 INFO [RS:0;jenkins-hbase20:34607] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34607,1680807373848; zookeeper connection closed. 2023-04-06 18:59:05,322 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@687cf671 rejected from java.util.concurrent.ThreadPoolExecutor@53638daf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2748] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,324 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7ffe37e5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7ffe37e5 2023-04-06 18:59:05,324 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): regionserver:34607-0x10087a33cf40001, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 18:59:05,325 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@33040f63 rejected from java.util.concurrent.ThreadPoolExecutor@53638daf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2748] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,325 INFO [Listener at localhost.localdomain/41219] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-06 18:59:05,422 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 18:59:05,422 INFO [M:0;jenkins-hbase20:45203] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,45203,1680807372735; zookeeper connection closed. 2023-04-06 18:59:05,422 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64730ab9 rejected from java.util.concurrent.ThreadPoolExecutor@278338c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4595] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,423 DEBUG [Listener at localhost.localdomain/41219-EventThread] zookeeper.ZKWatcher(600): master:45203-0x10087a33cf40000, quorum=127.0.0.1:63695, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 18:59:05,423 ERROR [Listener at localhost.localdomain/41219-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26017769 rejected from java.util.concurrent.ThreadPoolExecutor@278338c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4595] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 18:59:05,426 WARN [Listener at localhost.localdomain/41219] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-06 18:59:05,433 INFO [Listener at localhost.localdomain/41219] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-06 18:59:05,546 WARN [BP-1533696893-148.251.75.209-1680807370127 heartbeating to localhost.localdomain/127.0.0.1:34815] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-06 18:59:05,546 WARN [BP-1533696893-148.251.75.209-1680807370127 heartbeating to localhost.localdomain/127.0.0.1:34815] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1533696893-148.251.75.209-1680807370127 (Datanode Uuid c96f5453-3775-4f27-baac-cfdbfa27cff3) service to localhost.localdomain/127.0.0.1:34815 2023-04-06 18:59:05,549 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc/dfs/data/data1/current/BP-1533696893-148.251.75.209-1680807370127] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 18:59:05,550 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/2f68080c-38cc-135e-7afe-92cc5c03fb67/cluster_afb96881-d464-012e-2db4-39f6392db4dc/dfs/data/data2/current/BP-1533696893-148.251.75.209-1680807370127] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 18:59:05,583 INFO [Listener at localhost.localdomain/41219] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-06 18:59:05,747 INFO [Listener at localhost.localdomain/41219] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-06 18:59:05,898 INFO [Listener at localhost.localdomain/41219] hbase.HBaseTestingUtility(1293): Minicluster is down